Abstract
The neural codec language model (CLM) has demonstrated remarkable performance in text-to-speech (TTS) synthesis. However, troubled by "recency bias", CLM lacks sufficient attention to coarse-grained information at a higher temporal scale, often producing unnatural or even unintelligible speech. This work proposes CoFi-Speech, a coarse-to-fine CLM-TTS approach, employing multi-scale speech coding and generation to address this issue. We train a multi-scale neural codec, CoFi-Codec, to encode speech into a multi-scale discrete representation, comprising multiple token sequences with different time resolutions. Then, we propose CoFi-LM that can generate this representation in two modes: the single-LM-based chain-of-scale generation and the multiple-LM-based stack-of-scale generation. In experiments, CoFi-Speech significantly outperforms single-scale baseline systems on naturalness and speaker similarity in zero-shot TTS. The analysis of multi-scale coding demonstrates the effectiveness of CoFi-Codec in learning multi-scale discrete speech representations while keeping high-quality speech reconstruction. The coarse-to-fine multi-scale generation, especially for the stack-of-scale approach, is also validated as a crucial approach in pursuing a high-quality neural codec language model for TTS.
Introduction
Current state-of-the-art text-to-speech (TTS) models, such as VALL-E, treat speech synthesis as a "next-token prediction" task, analogous to large language models for text. However, speech sequences are significantly longer than text, leading to a phenomenon known as recency bias. In these models, the transformer often focuses excessively on the most recent tokens, losing track of global information like prosody, speaking style, and long-range phonetic structure. This results in speech that may sound locally coherent but becomes unstable, unnatural, or unintelligible over longer durations.
To address this, CoFi-Speech introduces a "coarse-to-fine" approach. By decomposing speech into multiple temporal scales—from broad prosodic structures to fine-grained acoustic details—it effectively mitigates recency bias and improves the stability of neural codec language models (CLMs).
Methodology
The CoFi-Speech framework consists of two primary components: a multi-scale neural codec and a hierarchical generative language model.
1. CoFi-Codec
Instead of encoding speech into a single sequence of tokens, CoFi-Codec decomposes speech into three distinct layers with varying temporal resolutions:
- Coarse Scale: 120ms frameshift (captures global prosody).
- Medium Scale: 40ms frameshift.
- Fine Scale: 20ms frameshift (captures detailed acoustics).
A critical challenge in multi-scale coding is "representation collapse," where the model ignores coarse layers in favor of fine-grained ones. To prevent this, the authors introduce Scale-Wise Nested Dropout (SWND). This technique randomly masks finer layers during training, forcing the model to rely on coarse layers for reconstruction.
2. CoFi-LM
The authors explored two architectures for generating these multi-scale tokens:
- Chain-of-Scale (CoS): A single LM predicts sequences in a serial chain (coarse, then medium, then fine).
- Stack-of-Scale (SoS): A cascade of multiple LMs where the hidden states of the coarse-scale LM guide the generation of the next, finer scale. This hierarchical approach allows each model to focus on a specific level of abstraction.
Experimental Results
The models were trained on the WenetSpeech4TTS (Basic) dataset, comprising approximately 7,000 hours of Chinese speech. Evaluation was conducted on an 860-utterance test set using metrics for naturalness (NMOS), speaker similarity (SMOS), and intelligibility (Character Error Rate - CER).
Performance Gains
CoFi-Speech (specifically the SoS version) significantly outperformed baseline systems:
- Naturalness: Achieved an NMOS of 4.42, surpassing VALL-E (3.46) and the industrial system CosyVoice (4.28).
- Speaker Similarity: Achieved an SMOS of 3.90, compared to VALL-E's 2.92.
- Intelligibility: The multi-scale approach showed a lower CER as more scales were added, indicating that coarse guidance helps the model maintain phonetic accuracy.
The Role of SWND
Quantitative analysis confirms that Scale-Wise Nested Dropout is essential for the effectiveness of the multi-scale representation. Without SWND, the codec fails to distribute information across scales properly, leading to higher distortion and error rates when reconstructing from coarse tokens.
Conclusion
CoFi-Speech demonstrates that a hierarchical, coarse-to-fine approach is vital for high-quality neural codec TTS. By explicitly modeling different temporal scales, the system overcomes the recency bias inherent in single-scale models. While the Stack-of-Scale approach adds architectural complexity, the resulting gains in prosodic stability and speaker similarity represent a significant step forward for zero-shot speech synthesis. Future work will explore extending this framework to multi-lingual datasets and integrating it with alternative generative backends like diffusion models.
Summary updated