TL;DR
Standard fine-tuning for one-shot voice cloning often causes autoregressive (AR) text-to-speech models to "collapse" into producing silent or near-empty audio. This paper demonstrates that the culprit is "off-policy" training on fixed targets and proposes two on-policy alternatives—GRPO and OPSD—that allow models to learn from their own generated speech, significantly boosting speaker similarity without sacrificing intelligibility.
What Problem Does This Paper Address?
Modern autoregressive TTS models (like LLaSA or Vall-E) treat speech as a sequence of discrete "codec tokens," much like how LLMs treat text. While these models are great at zero-shot voice cloning via in-context learning, researchers often want to "personalize" them further by fine-tuning on a specific speaker's voice.
However, when you only have a single short reference clip (one-shot), standard Supervised Fine-Tuning (SFT) fails spectacularly. The model essentially overfits to the "End of Sentence" (EOS) token. Because the training data always shows the reference audio followed immediately by an EOS token, the model learns that the best way to "imitate" the speaker is to stop talking as soon as the reference prompt ends. This leads to a massive spike in Word Error Rate (WER) and "empty" generations.
Methods
The authors argue that the fix is on-policy training: instead of training on a fixed, "off-policy" target, the model should learn from the trajectories it actually samples during training. They propose two methods:
- Group Relative Policy Optimization (GRPO): Borrowed from recent advances in LLM reasoning (like DeepSeek), GRPO samples a group of multiple speech outputs for a given text. It then uses a "Reward Function"—specifically, the cosine similarity between the speaker embeddings of the generated speech and the reference clip—to encourage the model to produce more similar-sounding voices.
- On-Policy Self-Distillation (OPSD): This is a faster, reward-free alternative. A "Student" model generates a speech trajectory based only on text. A "Teacher" model (the frozen original model) looks at that same trajectory but is given the "privileged" context of the reference audio. The student is then trained to match the teacher’s probability distribution (logits) over those specific samples.
Datasets
- Evaluation: The English subset of Seed-TTS-Eval, which provides reference utterances and ground-truth speech for evaluation.
- Augmentation: A pool of English sentences from LibriTTS was used to provide diverse text for the "Text-Augmented" versions of the algorithms.
- Base Model: All experiments were conducted using LLaSA-1B, a Llama-based AR TTS model.
Key Insights and Ideas
- The "EOS Collapse" is an Objective Problem: The failure of SFT isn't just simple overfitting; it's a fundamental flaw in the training objective. Even using LoRA (Low-Rank Adaptation) doesn't fix it because the model is still being told that "silence follows the prompt."
- Diversity Isn't Enough: The authors tested "Zero-Shot Distilled SFT," where they generated many diverse pseudo-pairs to train on. While this stopped the model from going silent, it didn't actually improve speaker similarity much.
- On-Policy is the "Aha" Moment: By training on its own sampled outputs, the model learns to navigate the transition from the reference prompt to new speech without crashing into an EOS token.
- Efficiency Trade-offs: GRPO provides the best results but is computationally expensive because it requires full audio synthesis (vocoding) and a separate reward model for every training step. OPSD is roughly 8x faster because it operates entirely in the token/logit space.
Key Results
The on-policy methods outperformed both the zero-shot baseline and traditional SFT across the board.
| Method | WER (%) ↓ | Speaker Sim (SIM) ↑ | Speed (vs SFT) |
|---|---|---|---|
| Zero-shot (Baseline) | 3.7 | 0.574 | - |
| SFT (Traditional) | 18.2 | 0.536 | 1.0x |
| OPSD-TA (Proposed) | 3.7 | 0.609 | 5.8x |
| GRPO (Proposed) | 3.0 | 0.647 | 45.2x |
- Intelligibility: GRPO actually improved the Word Error Rate compared to the zero-shot baseline (3.0% vs 3.7%), while SFT nearly quintupled it.
- Similarity: Both methods significantly boosted speaker similarity (SIM), with GRPO reaching the highest scores.
- Naturalness: Human evaluators (NMOS) rated the on-policy methods as comparable in naturalness to the ground truth and zero-shot synthesis.
Limitations and Future Work
- Compute Cost: While OPSD is faster than GRPO, it is still ~6x slower than basic SFT due to the need for autoregressive sampling during the training loop.
- Teacher Dependence: OPSD's performance is capped by the "privileged" teacher's zero-shot capability. If the base model is poor at zero-shot cloning, the distillation will be less effective.
- Language Scope: The study focused exclusively on English; the authors suggest that exploring multi-lingual one-shot personalization is a logical next step.
Summary updated