Problem
A storybook pipeline has to do more than generate one attractive image. It must translate narrative text into usable scene inputs, retain character identity over multiple pages, expose targeted corrections, and move assets reliably between generation, editing, and delivery stages. The work focused on integrating those stages into workflows that people could inspect and correct.
Constraints
- Characters should remain recognizable across pages with different poses, framing, lighting, and backgrounds.
- Segmentation around hair, occlusion, clothing, and overlapping characters is inherently imperfect.
- Editors need to correct masks instead of being locked into automatic model output.
- Generation and post-processing stages can fail independently and produce incomplete assets.
- Model-serving endpoints need stable input, output, and asset-lifecycle conventions.
- Region-based edits should avoid altering unrelated parts of an illustration.
Architecture
- 1
Story decomposition
Transforms narrative text into scene-level inputs and per-page generation context.
- 2
Reference-conditioned generation
Uses character references and diffusion workflows to improve multi-page visual consistency.
- 3
Cross-image character matching
Identifies the selected character across generated pages before localized editing begins.
- 4
Editable segmentation
Generates an initial mask while preserving a manual correction step for ambiguous boundaries.
- 5
Region-based editing
Applies face replacement, inpainting, recolouring, and related edits to the selected region.
- 6
FastAPI serving layer
Standardizes workflow inputs, outputs, asset paths, and error handling across model stages.
Design decisions
Break the workflow into inspectable stages
Scene preparation, generation, matching, segmentation, and editing fail for different reasons. Keeping them distinct makes errors diagnosable and allows a successful upstream asset to be reused instead of regenerating everything.
Use automatic masks with manual correction
Segmentation accelerates the common case, but edge quality varies around occlusion and fine detail. Editable masks keep the workflow useful when automatic output is not precise enough.
Treat character identity as cross-image state
A page-local edit is insufficient when the same character appears throughout a book. Matching the user's selection across pages makes batch editing possible while retaining per-page correction.
Standardize assets at the API boundary
Explicit inputs, outputs, identifiers, and asset handling reduce coupling between UI actions and ComfyUI or model-specific execution details.
Correctness and testing
- Reviewed intermediate scene inputs and generated pages rather than evaluating only the final composite output.
- Exercised editing workflows across multiple pages and character selections.
- Debugged inference and post-processing failures independently at their workflow boundaries.
- Validated standardized API inputs, outputs, and asset references across generation and editing stages.
Failure cases
- Character identity can drift when references, pose, or scene composition change substantially.
- Occlusion and fine boundaries can produce weak automatic masks that require correction.
- Inpainting can modify texture, lighting, or identity outside the intended semantic change.
- A successful model response can still fail during asset transfer or post-processing.
Results
- Integrated story decomposition and multi-page illustration generation into a single usable workflow.
- Enabled selected characters to be found and edited across multiple generated images.
- Combined automatic segmentation with editor-controlled mask correction.
- Exposed generation and editing stages through FastAPI model-serving endpoints.
Limitations
The system is intentionally described within the boundaries supported by its implementation and evidence.
- This case study describes my workflow ownership; it does not claim ownership of the full NiftyBooks product.
- No unsupported accuracy, latency, or scale figures are presented.
- Consistent generation remains probabilistic and needs human review.
- Manual correction remains important for high-detail masks and identity-sensitive edits.