TL;DR
CoFi-Speech introduces a "coarse-to-fine" approach to neural codec language models (CLMs) for text-to-speech. By decomposing speech into multiple temporal scales—from broad prosodic structures to fine-grained acoustic details—it effectively solves the "recency bias" that often makes standard CLMs produce unstable or unnatural speech.
What Problem Does This Paper Address?
Current state-of-the-art text-to-speech (TTS) models, like VALL-E, treat speech synthesis as a "next-token prediction" task, similar to how LLMs handle text. However, speech sequences are significantly longer than text sequences. This length leads to recency bias: the model focuses too much on the most recent tokens and loses track of global information like prosody, speaking style, and long-range phonetic structure.
The result? Speech that might sound okay locally but becomes unstable, unnatural, or even unintelligible over longer durations. While some researchers try to fix this with attention constraints or by simply using larger "frameshifts" (shorter sequences), they often sacrifice fine-grained detail and speaker similarity.
Methods
The authors propose CoFi-Speech, which consists of two main components:
1. CoFi-Codec (The Encoder/Decoder)
Instead of encoding speech into a single sequence of tokens, this multi-scale neural codec decomposes speech into three distinct layers with different temporal resolutions:
- Coarse Scale: 120ms frameshift (captures global prosody).
- Medium Scale: 40ms frameshift.
- Fine Scale: 20ms frameshift (captures detailed acoustics).
To prevent the model from ignoring the coarse layers (a common issue called "representation collapse"), they introduce Scale-Wise Nested Dropout (SWND). This technique randomly masks finer layers during training, forcing the model to rely on the coarse layers for reconstruction.
2. CoFi-LM (The Generative Model)
The authors explored two ways to generate these multi-scale tokens:
- Chain-of-Scale (CoS): A single Large Language Model (LM) predicts the sequences in a "chain" (coarse tokens first, then medium, then fine).
- Stack-of-Scale (SoS): A cascade of multiple LMs. The first LM generates the coarse sequence. Its hidden states are then passed to the next LM to generate the finer details. This "hierarchical" approach allows each model to focus on a specific level of abstraction.
Datasets
The models were trained on WenetSpeech4TTS (Basic), a large-scale Chinese dataset containing approximately 7,000 hours of speech.
- Evaluation: An 860-utterance test set with diverse speaking styles.
- Metrics: Naturalness (NMOS), Speaker Similarity (SMOS), Character Error Rate (CER), and Mel-cepstrum distortion (MCD).
Key Insights and Ideas
- Solving Recency Bias via Hierarchy: The most significant "aha moment" is the attention visualization. In standard single-scale LMs, the attention is a narrow diagonal line (focusing only on the immediate past). In CoFi-Speech, the coarse-scale LM shows a much broader attention map, proving it is actually "looking" at the entire context to plan the speech's prosody.
- The Power of the "Stack": The Stack-of-Scale (SoS) approach outperformed the Chain-of-Scale (CoS). By separating the scales into different models, the system avoids the computational burden of extremely long sequences while allowing for better alignment between coarse and fine features.
- SWND is Essential: Without Scale-Wise Nested Dropout, the codec tends to put all the "information" into the fine-grained tokens, making the coarse tokens useless for guidance.
Key Results
CoFi-Speech (specifically the SoS version) significantly outperformed baseline systems:
- Naturalness (NMOS): Achieved a score of 4.42, beating VALL-E (3.46) and even the high-performing industrial system CosyVoice (4.28).
- Speaker Similarity (SMOS): Achieved 3.90, compared to VALL-E's 2.92 and SoCodec-TTS's 3.51.
- Intelligibility: In zero-shot TTS, the multi-scale approach showed a lower Character Error Rate (CER) as more scales were added, indicating that coarse guidance helps the model stay "on track" with the text.
Limitations and Future Work
- Inference Complexity: The Stack-of-Scale approach requires running multiple LMs in a cascade. While this avoids the "long sequence" problem for any single transformer, it adds architectural complexity.
- Language Scope: The study focused on a Chinese dataset. While the principles of multi-scale speech are universal, testing on more diverse, multi-lingual datasets would further validate the approach.
- Future Directions: The authors suggest that this coarse-to-fine framework could be extended to even more scales or integrated with different types of generative models (like diffusion-based codecs) to further push the boundaries of speech quality.
Summary updated