NiftyBooks illustration system

The hard part was not generating one good image. A storybook also has to keep a character recognizable across pages, let an editor fix only the part that is wrong, and move assets reliably between generation, editing, and delivery. My work connected those stages into workflows people could inspect and correct.

• A character should stay recognizable when pose, framing, lighting, and background change from page to page.
• Hair, clothing, occlusion, and overlapping characters make automatic segmentation imperfect.
• An editor needs to be able to correct a mask rather than accept the model output as final.
• Generation and post-processing fail independently, sometimes leaving incomplete assets behind.
• The serving layer needs stable conventions for inputs, outputs, and asset lifecycles.
• A region-based edit should leave the rest of the illustration alone.

1. 1

   Story decomposition

   The first stage turns narrative text into scene inputs and per-page generation context.

2. 2

   Reference-conditioned generation

   Character references condition the diffusion workflows used across the pages.

3. 3

   Cross-image character matching

   The editing path finds the selected character across the generated pages before making a local change.

4. 4

   Editable segmentation

   Segmentation provides the first mask, and the editor can correct ambiguous boundaries by hand.

5. 5

   Region-based editing

   Face replacement, inpainting, recolouring, and related tools operate on the selected region.

6. 6

   FastAPI serving layer

   FastAPI gives the model stages a common contract for inputs, outputs, asset paths, and errors.

• I reviewed intermediate scene inputs and generated pages, not only the final composite output.
• I exercised the editing workflow across multiple pages and character selections.
• Inference and post-processing failures were debugged at their own workflow boundaries.
• API inputs, outputs, and asset references were checked across generation and editing stages.
• I recorded the masked-editing latency before and after replacing its generative stage.

• Character identity can drift when references, pose, or scene composition change substantially.
• Occlusion and fine boundaries can weaken an automatic mask enough to require manual correction.
• Inpainting can alter texture, lighting, or identity beyond the intended change.
• A model can return successfully and the asset can still fail during transfer or post-processing.
• A poor or incomplete mask still produces a poor OpenCV edit.

• Story decomposition and multi-page illustration generation now run as one usable workflow.
• An editor can find a selected character and edit it across multiple generated images.
• Automatic segmentation provides a starting mask that the editor can correct.
• The profiled masked edit now uses OpenCV instead of an unnecessary generative operation.
• FastAPI endpoints expose the generation and editing stages.

One masked edit took more than 90 seconds. Profiling showed that a generative stage dominated the wait even though the transformation and editable mask were already known; the operation did not need to invent new image content.

• I replaced the expensive stage with a mask-constrained OpenCV path.
• The surrounding FastAPI contract for inputs, outputs, and assets did not change.
• End-to-end latency for the tested operation fell from more than 90 seconds to approximately 5–6.5 seconds.
• That number is for this masked edit, not production p95 or the complete multi-page generation pipeline.
• An edit that must synthesize missing structure still uses a generative operation.

• This case study covers the workflows I owned, not the full NiftyBooks product.
• Consistent generation is still probabilistic and needs human review.
• High-detail masks and identity-sensitive edits still need manual correction.