Beyond Fixed Targets: On-Policy One-Shot Speaker Personalization for Autoregressive Text-To-Speech Models

Sangkwon Park, Che Hyun Lee, Heeseung Kim, Sungroh YoonarXiv 2026 · arXiv

TL;DR

Standard fine-tuning for one-shot voice cloning often causes autoregressive (AR) text-to-speech models to "collapse" into producing silent or near-empty audio. This paper demonstrates that the culprit is "off-policy" training on fixed targets and proposes two on-policy alternatives—GRPO and OPSD—that allow models to learn from their own generated speech, significantly boosting speaker similarity without sacrificing intelligibility.

What Problem Does This Paper Address?

Modern autoregressive TTS models (like LLaSA or Vall-E) treat speech as a sequence of discrete "codec tokens," much like how LLMs treat text. While these models are great at zero-shot voice cloning via in-context learning, researchers often want to "personalize" them further by fine-tuning on a specific speaker's voice.

However, when you only have a single short reference clip (one-shot), standard Supervised Fine-Tuning (SFT) fails spectacularly. The model essentially overfits to the "End of Sentence" (EOS) token. Because the training data always shows the reference audio followed immediately by an EOS token, the model learns that the best way to "imitate" the speaker is to stop talking as soon as the reference prompt ends. This leads to a massive spike in Word Error Rate (WER) and "empty" generations.

Fig. 2. Ratio of empty generations per training step.

Methods

The authors argue that the fix is on-policy training: instead of training on a fixed, "off-policy" target, the model should learn from the trajectories it actually samples during training. They propose two methods:

  1. Group Relative Policy Optimization (GRPO): Borrowed from recent advances in LLM reasoning (like DeepSeek), GRPO samples a group of multiple speech outputs for a given text. It then uses a "Reward Function"—specifically, the cosine similarity between the speaker embeddings of the generated speech and the reference clip—to encourage the model to produce more similar-sounding voices.
  2. On-Policy Self-Distillation (OPSD): This is a faster, reward-free alternative. A "Student" model generates a speech trajectory based only on text. A "Teacher" model (the frozen original model) looks at that same trajectory but is given the "privileged" context of the reference audio. The student is then trained to match the teacher’s probability distribution (logits) over those specific samples.
Fig. 1. Overview of off-policy SFT and proposed on-policy methods for one-shot personalization of autoregressive TTS.

Datasets

  • Evaluation: The English subset of Seed-TTS-Eval, which provides reference utterances and ground-truth speech for evaluation.
  • Augmentation: A pool of English sentences from LibriTTS was used to provide diverse text for the "Text-Augmented" versions of the algorithms.
  • Base Model: All experiments were conducted using LLaSA-1B, a Llama-based AR TTS model.

Key Insights and Ideas

  • The "EOS Collapse" is an Objective Problem: The failure of SFT isn't just simple overfitting; it's a fundamental flaw in the training objective. Even using LoRA (Low-Rank Adaptation) doesn't fix it because the model is still being told that "silence follows the prompt."
  • Diversity Isn't Enough: The authors tested "Zero-Shot Distilled SFT," where they generated many diverse pseudo-pairs to train on. While this stopped the model from going silent, it didn't actually improve speaker similarity much.
  • On-Policy is the "Aha" Moment: By training on its own sampled outputs, the model learns to navigate the transition from the reference prompt to new speech without crashing into an EOS token.
  • Efficiency Trade-offs: GRPO provides the best results but is computationally expensive because it requires full audio synthesis (vocoding) and a separate reward model for every training step. OPSD is roughly 8x faster because it operates entirely in the token/logit space.

Key Results

The on-policy methods outperformed both the zero-shot baseline and traditional SFT across the board.

Method WER (%) ↓ Speaker Sim (SIM) ↑ Speed (vs SFT)
Zero-shot (Baseline) 3.7 0.574 -
SFT (Traditional) 18.2 0.536 1.0x
OPSD-TA (Proposed) 3.7 0.609 5.8x
GRPO (Proposed) 3.0 0.647 45.2x
  • Intelligibility: GRPO actually improved the Word Error Rate compared to the zero-shot baseline (3.0% vs 3.7%), while SFT nearly quintupled it.
  • Similarity: Both methods significantly boosted speaker similarity (SIM), with GRPO reaching the highest scores.
  • Naturalness: Human evaluators (NMOS) rated the on-policy methods as comparable in naturalness to the ground truth and zero-shot synthesis.
Table 1. One-shot personalization results with LLaSA-1B on Seed-TTS-Eval. SFT-ZSD denotes Zero-Shot Distilled SFT; t denotes the adaptation time measured per training step in seconds, and vs. SFT reports the ratio of t relative to SFT.

Limitations and Future Work

  • Compute Cost: While OPSD is faster than GRPO, it is still ~6x slower than basic SFT due to the need for autoregressive sampling during the training loop.
  • Teacher Dependence: OPSD's performance is capped by the "privileged" teacher's zero-shot capability. If the base model is poor at zero-shot cloning, the distillation will be less effective.
  • Language Scope: The study focused exclusively on English; the authors suggest that exploring multi-lingual one-shot personalization is a logical next step.

Summary updated

Abstract

One-shot personalization of autoregressive (AR) text-to-speech (TTS) models remains challenging because supervised fine-tuning (SFT) on a single reference pair rapidly degrades generation quality. We identify fixed, off-policy supervision as a key cause of failure in one-shot SFT for AR TTS, leading to blank or near-empty outputs at inference. Diversifying the training data with zero-shot pseudo-pairs eliminates this failure but yields little improvement in speaker similarity and increases word error rate because the targets remain fixed and teacher-forced. We therefore propose two on-policy approaches for one-shot speaker personalization: Group Relative Policy Optimization (GRPO) with a speaker-similarity reward and On-Policy Self-Distillation (OPSD) with soft supervision from a reference-conditioned teacher. Both on-policy methods improve speaker similarity while preserving intelligibility, with different trade-offs: GRPO achieves the strongest overall performance, whereas OPSD enables substantially faster adaptation without external reward evaluation. Further analysis confirms that these gains stem from on-policy supervision rather than target diversity, establishing on-policy training as an effective framework for one-shot AR TTS personalization.
BibTeX
@misc{park2026beyond,
  author = {Sangkwon Park and Che Hyun Lee and Heeseung Kim and Sungroh Yoon},
  title = {{Beyond Fixed Targets: On-Policy One-Shot Speaker Personalization for Autoregressive Text-To-Speech Models}},
  year = {2026},
  eprint = {2610.17838},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.17838},
}
Figure