Abstract
Modern language model (LM)-based text-to-speech (TTS) systems generate highly natural speech but are prone to "hallucinations"—such as word repetitions and mispronunciations—stemming from the randomness of standard sampling techniques. This paper introduces Temporal Repetition Aware Diverse Beam Search (TRAD-BS), a modified decoding strategy designed to stabilize TTS output. By applying penalized beam search, TRAD-BS effectively mitigates temporal and diversity collapse. Experimental results on state-of-the-art models like VoiceCraft and CosyVoice demonstrate significant reductions in word error rates (WER) and improved speaker consistency, with strong human preference over traditional sampling methods.
Introduction
The paradigm of Text-to-Speech (TTS) has recently shifted toward Large Language Model (LLM) architectures, such as VALL-E and XTTS. These systems represent speech as sequences of discrete "tokens" or quantized audio units. To generate expressive and human-like speech, these models typically rely on sampling-based decoding (e.g., top-p or top-k).
However, the low information density of speech compared to text presents a challenge: a short sentence can translate into hundreds of tokens. In these extended sequences, stochastic sampling often leads to instability, causing the model to skip words, repeat phrases, or lose the speaker's characteristic tone. While standard Beam Search is a reliable deterministic alternative for text, it fails in the audio domain due to "temporal collapse" (looping silence or static) and "diversity collapse" (identical beams). This research addresses these failures to make LM-based TTS more reliable.
Methodology: TRAD-BS
The proposed Temporal Repetition Aware Diverse Beam Search (TRAD-BS) guides the decoding process away from degenerate paths by implementing two specific penalties:
- Temporal Repetition Penalty ($\alpha$): This penalizes tokens that have appeared recently within a specific window of the current beam, preventing "stuck record" artifacts.
- Beam-wise Repetition Penalty ($\beta$): This penalizes a beam if it attempts to generate a token already selected by a different beam at the same time step, forcing the search to explore diverse phonetic paths.
The workflow involves generating tokens autoregressively while modifying log-probabilities with $\alpha$ and $\beta$. Crucially, once decoding is complete, the beams are re-ranked using their original, unpenalized probabilities to ensure the final output remains mathematically consistent with the model's learned distribution.
Experimental Setup
The researchers validated TRAD-BS using two zero-shot TTS models:
- VoiceCraft (English): Trained on 9,000 hours of GigaSpeech audio.
- CosyVoice (Chinese): Trained on a massive 170,000-hour multilingual dataset.
Performance was benchmarked against the standard sampling configurations provided by the original model authors using the SeedTTS-EN and SeedTTS-CN test sets.
Results and Discussion
TRAD-BS yielded consistent improvements across objective and subjective metrics. For VoiceCraft, the Word Error Rate (WER) dropped from 5.71% with sampling to 3.73% with TRAD-BS. Speaker Similarity (SS) scores also improved, indicating better maintenance of voice identity.
Human listeners showed a strong preference for TRAD-BS in blind tests, with preference rates of 71.95% for VoiceCraft and 62.30% for CosyVoice. The method effectively eliminated the "silence trap"—where high-probability silence tokens cause temporal collapse—by recognizing that the most mathematically probable path is not always acoustically correct.
Conclusion
TRAD-BS provides a robust solution to the instability of LM-based TTS systems. While it significantly improves accuracy and stability, there is a minor trade-off in prosodic expressiveness compared to "lucky" stochastic samples. Future research will explore hybrid approaches to combine the robustness of TRAD-BS with the rhythmic variety of sampling-based methods.
Summary updated