TL;DR
While language model (LM)-based text-to-speech (TTS) systems produce incredibly natural voices, they often suffer from "hallucinations" like word repetitions or mispronunciations due to the randomness of standard sampling techniques. This paper introduces TRAD-BS (Temporal Repetition Aware Diverse Beam Search), a modified decoding strategy that replaces unstable sampling with a penalized beam search, significantly reducing error rates and improving speaker consistency across state-of-the-art models like VoiceCraft and CosyVoice.
What Problem Does This Paper Address?
Modern TTS has shifted toward Large Language Model paradigms (e.g., VALL-E, XTTS). These models treat speech as a sequence of discrete "tokens" (quantized audio units), much like how GPT treats words. To generate speech, these models typically use sampling-based decoding (like top-p or top-k), which introduces randomness to ensure the output sounds expressive and human-like.
However, speech has a much lower information density than text. A ten-word sentence might translate into hundreds of speech tokens. In such long sequences, the randomness of sampling often leads to "instability": the model might skip words, repeat phrases, or lose the speaker's characteristic tone halfway through.
Standard Beam Search—the go-to deterministic alternative in text generation—fails spectacularly here. When applied to speech tokens, it usually suffers from "temporal collapse" (getting stuck in a loop of silence or static) or "diversity collapse" (where all candidate "beams" become identical). This paper aims to fix beam search so it can finally be used to make TTS more reliable.
Methods
The authors propose TRAD-BS (Temporal Repetition Aware Diverse Beam Search). The core idea is to guide the beam search away from the "easy" but degenerate paths (like silence) by applying two specific penalties during the decoding process:
- Temporal Repetition Penalty ($\alpha$): This penalizes the model if it tries to generate a token that has appeared recently within a specific window of the current beam. This prevents the "stuck record" effect where the model repeats a single sound or artifact.
- Beam-wise Repetition Penalty ($\beta$): This penalizes a beam if it tries to generate a token that has already been selected by a different beam at the same time step. This forces the search to explore diverse phonetic paths rather than having all five beams converge on the exact same sequence.
The Workflow: * Tokens are generated autoregressively. * The log-probabilities are modified by $\alpha$ and $\beta$ to discourage repetition. * Once decoding is finished, the beams are re-ranked using their original (unpenalized) probabilities to ensure the final selection is still mathematically likely according to the model.
Datasets
The researchers validated their approach using two high-performing, zero-shot TTS models: VoiceCraft (English) and CosyVoice (Chinese).
- Training Data: VoiceCraft was trained on GigaSpeech (9,000 hours of audiobooks/podcasts). CosyVoice was trained on a massive multilingual dataset of 170,000 hours.
- Evaluation Sets:
- English: A subset of GigaSpeech and the SeedTTS-EN test set.
- Chinese: An internal "expressive" dataset and SeedTTS-CN.
- Benchmarks: They compared TRAD-BS against the standard sampling configurations released by the original model authors.
Key Insights and Ideas
- The "Silence" Trap: In many neural audio codecs, "silence" or "background noise" is represented by highly predictable, high-probability tokens. Standard beam search greedily gravitates toward these, leading to the "temporal collapse" mentioned earlier. TRAD-BS recognizes that the most probable mathematical path isn't always the most correct acoustic path.
- Diversity as a Safety Net: By forcing beams to be different, the algorithm essentially creates five different "interpretations" of the text. Even if the top beam fails, the diversity penalty ensures that the other beams provide viable alternatives.
- Post-hoc Re-ranking: A crucial trick in the paper is using the penalties only to find candidates, but using the original model scores to rank them at the end. This maintains the model's learned distribution while avoiding the local minima of repetitive artifacts.
Key Results
The improvements were consistent across both objective metrics and human listening tests:
- Error Rates: For VoiceCraft on the GigaSpeech set, the Word Error Rate (WER) dropped from 5.71% (sampling) to 3.73% (TRAD-BS). Similar gains were seen in Chinese Character Error Rates (CER).
- Speaker Consistency: TRAD-BS showed higher Speaker Similarity (SS) scores, meaning the voice stayed "in character" more reliably throughout the clip.
- Subjective Preference: In "blind" tests, human listeners preferred TRAD-BS over the baseline sampling method by a wide margin: 71.95% preference for VoiceCraft and 62.30% for CosyVoice.
- Stability: TRAD-BS effectively eliminated the "hallucinations" (random noise or deleted words) that frequently plague long-form synthesis in LM-based models.
Limitations and Future Work
While TRAD-BS wins on stability and accuracy, the authors note a trade-off: Expressiveness.
Sampling-based methods, despite their flaws, produce more varied prosody (the "rhythm" and "melody" of speech). Because TRAD-BS is a maximization-based approach, it can sometimes result in speech that sounds slightly more "stable" but less "alive" or emotionally varied than a lucky sample from a top-p distribution.
Future work will likely focus on finding a "middle ground"—perhaps a hybrid approach that maintains the robustness of TRAD-BS while re-introducing some of the prosodic flair found in stochastic sampling.
Summary updated