AI Voice Lab
A chain of local text-to-speech engines with per-character voices, voice cloning from a reference clip, and a fallback for every line.
- Status
- Experiment
- Role
- Developer
- Year
- 2026
- Type
- Local speech synthesis
Where it stands
Local only. This is speech synthesis, not a conversational agent, and no hosted demo exists.
- Engines
- Kokoro, GPT-SoVITS, Qwen TTS, system voice
- Runs on
- CPU for the default path
- Used by
- Video Director
There is no public link for this project.
The problem
Voicing a script with several characters needs distinct, consistent voices and has to finish even when one engine refuses a line.
Goals and constraints
It had to
- One distinct voice per character.
- A cloned voice when a reference recording exists.
- A complete audio track every time, with gaps reported.
Within these limits
- Each engine has its own dependency set.
- The GPU is usually busy with image or video work.
My part
Integration and orchestration around open-source engines. I did not train the models.
- Wrote the engine chain and the per-line fallback.
- Wrapped each engine as a worker in its own environment.
- Wired voice assignment into the production pipeline's configuration.
What it is
Lines are sent to the first available engine. A line that fails is passed down the chain on its own; the rest of the track is unaffected. Silent placeholders are retried on the next run.
- Fallback per line
- Kokoro, then GPT-SoVITS, then the system voice, then silence. An engine choking on punctuation does not cost the whole scene.
- Voice cloning
- A character with a reference clip and its transcript is voiced by GPT-SoVITS in that voice.
- Isolated workers
- Each engine runs in its own virtual environment and is started on demand.
Decisions and trade-offs
01CPU first
The default engine runs on the CPU so speech never competes with the video model for memory.
The cost. Slower than GPU synthesis.
02Silence is a valid result
A missing line becomes a labelled silent placeholder, not a failed run.
The cost. The report has to be read to know the track is incomplete.
How it is built
- Worker processes with a small request protocol.
- FFmpeg for assembly and timing.
Python · Kokoro-82M · GPT-SoVITS · Qwen TTS · Windows SAPI · FFmpeg
Hard problems
Without a real reference clip, the cloning engine copies the system voice and sounds robotic.
Cloning is only used when a reference and its transcript are supplied. Otherwise the natural built-in voices are preferred.
Limits and open work
- Source pages do not say what a character sounds like, so voices are assigned by configuration.
- No audio samples are published here.
Need something like this built?
Tell me what you are making. I will tell you how I would approach it, and whether I am the right person.
Start a conversation