Video Director
A local pipeline that takes comic pages to a voiced, subtitled episode, with a human review gate wherever a model can be confidently wrong.
- Status
- Experiment
- Role
- Designer and developer
- Year
- 2026
- Type
- AI-assisted animation and manga production pipeline
Where it stands
Runs locally on one laptop GPU. It has produced one complete 58.9-second run from an original page. No finished episode has been validated, and the film itself is not published.
- Pipeline
- 13 resumable stages, 6 review checkpoints
- Workflows
- 24 ComfyUI workflows
- Hardware
- One laptop GPU, fully local
There is no public link for this project.
The problem
Turning a comic page into animation is a long chain of steps, and each step uses a model that is sometimes wrong in ways that look fine at a glance: a merged panel, an invented prop, a face that drifts by frame 30.
Chaining those models end to end and trusting the output produces something that plays and is quietly wrong. The interesting problem is where to put the checks.
Goals and constraints
It had to
- Go from pages to an episode file with subtitles and production metadata.
- Never change a line of dialogue. The source text is copied, not regenerated.
- Stop for a person at the stages where automation cannot be trusted.
- Survive interruptions: a run can stop anywhere and resume.
- Label the result for what it is.
Within these limits
- Everything runs on a single laptop GPU, so only one model family can be loaded at a time.
- Heat and power are real limits. Language-model inference is treated as GPU load, with the same checks as video generation.
- Open models come with mixed licences, which restricts what can be shown.
My part
A personal research project. The pipeline, the review tooling and the benchmarks are mine; the models it drives are third-party and open-weight.
- Built the bridge that drives ComfyUI over its HTTP API and verifies every output file.
- Built the thirteen-stage manga pipeline with artifacts, checkpoints and downstream invalidation.
- Built panel detection, text removal, shot planning and the assembly step that performs camera moves.
- Built the review gate, the calibration report and a local web studio for one-click runs.
- Benchmarked seven local language models on the director's tasks and chose the architecture from the results.
What it is
Pages go in as a PDF or images. The pipeline finds panels and reading order, reads the text, writes a screenplay, plans one shot per panel, removes lettering, optionally colourises, builds plates, animates what it is allowed to, voices the lines, assembles the film and reports on quality.
At six of those stages it stops and waits. A person reviews the artifact, edits it if needed, and approves. Editing an artifact re-runs everything downstream of it.
- Resumable stage machine
- Each stage writes an artifact and re-runs only when something upstream changed. Failures are recorded per stage and retried on the next run.
- Verbatim dialogue
- A fidelity check fails the screenplay stage if a line is lost or altered. The language model's schema has no field for dialogue text, so it cannot rewrite one.
- Review gate
- Every clip review answers three questions: does it match the panel, does it match the prompt beat by beat, and does it need more movement. A rejection can carry a direction that is written into the next attempt.
- Resource planning
- A pre-flight check downgrades or rejects a clip that cannot fit in memory before any model loads. On an out-of-memory error it steps down a ladder instead of retrying blindly.
- Voice chain
- Kokoro first, then GPT-SoVITS, then the system voice, then silence. A line one engine rejects falls back for that line only.
- Honest labels
- The export is called animation only when every shot is a generated clip. Otherwise it is labelled a motion comic, or mixed with the exact count.
Decisions and trade-offs
01Rules for anything with one right answer
Scene grouping, panel ids, durations, camera defaults and backend choice stay in code. The language model only adds direction: movement, beats, end pose, voice direction. This came from a benchmark rather than a preference.
The cost. Less magic. The model could do more, but none of the seven tested grouped scenes correctly, and the top four were within 0.03 of each other on the rest.
02Automated quality checks are advisory
Frame-similarity metrics pass a static or partly melted picture, and the vision model tested rejected none of the bad pilot clips. So automation only catches broken files. A person decides the rest.
The cost. A run is not unattended. That is the honest state of the tools.
03The camera moves in assembly, not in the model
The video model ignored a planned slow dolly in every clip, and prompts for camera motion made it lurch. Clips are now always requested with a static camera, and assembly performs the push, pan or shake.
The cost. Camera moves are 2D operations on a clip, so there is no true parallax.
04Blocked means blocked
When a stage cannot run, because a workflow is missing or a safety interlock refuses, the pipeline stops with instructions or falls back to a labelled animatic. It never silently substitutes something else under the same name.
The cost. More stopped runs, fewer surprises in the output.
Try it
A model of the stage machine. No AI runs here. It shows the control flow: where the pipeline stops for a person, and what an edit sends back to be redone.
Nothing has run yet.
Ingest
Stage 1 of 13Reads a PDF or a folder of images into ordered pages.
- Writes
- Page images in reading order
- When it goes wrong
- Unsupported page encodings stop with a request to export the pages as PNG or JPG.
- Stops for review
- No
- Times run
- 0
Available once this stage has finished.
How it is built
- A Python package with a standard-library core, driving an existing ComfyUI server through its API with API-format workflows and small parameter maps.
- Stage state and artifacts are written atomically, with retries, after a real run exposed a file-replace race on Windows.
- Voices run as worker processes in their own virtual environments so their dependencies never collide with the pipeline's.
- A local web studio bound to localhost handles generate, pause, resume, per-shot re-render and a panel-box editor.
Python · ComfyUI API · Wan 2.2 · Z-Image Turbo · Ollama · Qwen · Kokoro · GPT-SoVITS · FFmpeg · ONNX Runtime
Hard problems
Panel detection found 3 panels on a page that has 12, because the panels touch and one gutter is slanted.
Higher analysis resolution, a smaller minimum gutter, a search for leaning gutters, and tolerance for one crossing in a wide band. On that page it now finds 12 of 12 within 8 pixels of the hand-set boxes, with unchanged counts on the other test pages.
At low resolution the video model lost a character's face in wide shots.
Measured it: around 40 pixels of face height failed and around 55 held. Start frames are now cropped to fill the frame, centred on the largest drawn face.
A custom node broke against a newer ComfyUI core and failed before sampling.
Found the open upstream issues, applied the correct fix as a local patch, and documented why the shorter workaround silently drops an activation function.
Removing lettering from pure noise made the inpainting model paint a new speech balloon and a second face.
A pre-fill pass plus partial denoising strength, so the model repairs a region instead of inventing one.
What can be verified
- first complete run, 20 lines voicedProject log, 11 October 2026. One original page, 12 panels.
- 58.9 s
- generated shots approved on the first attemptSame run. The seventh used its three attempts and fell back to a still.
- 6 of 7
- local language models benchmarked as directorNine tasks each, scored and logged with timing, memory and temperature.
- 7
Limits and open work
- Not validated. Thresholds were tuned on a handful of pages and are unmeasured elsewhere.
- Emptied speech balloons, held last frames and soft upscaled panels are all still visible in the output.
- One colouriser model has no licence file, so its results are for local testing only and are not shown here.
- The stills on this page come from that one run. The film itself is not published.
Need something like this built?
Tell me what you are making. I will tell you how I would approach it, and whether I am the right person.
Start a conversation