TL;DR
This paper introduces CMG-TTS, a novel two-stage text-to-speech (TTS) model that significantly speeds up diffusion-based synthesis by employing an "active shallow diffusion mechanism." By training a basic acoustic model first to provide prior knowledge, CMG-TTS achieves high-quality speech with just one denoising step, outperforming existing diffusion TTS models in real-time factor (RTF).
What Problem Does This Paper Address?
Traditional diffusion models, while powerful for generative tasks like speech synthesis, suffer from slow inference times due to the need for hundreds of iterative denoising steps. This makes real-time TTS applications challenging. Existing diffusion-based TTS models have attempted to address this by reducing the number of denoising steps, but often at the cost of speech quality or by introducing complex alignment mechanisms. The paper aims to bridge this gap by developing a TTS model that is both high-quality and efficient for real-time inference, specifically by optimizing the diffusion process.
Methods
The core innovation is the Cascaded MixGAN-TTS (CMG-TTS), a two-stage architecture that leverages an active shallow diffusion mechanism.
- Stage 1: Basic Acoustic Model: This stage trains a foundational acoustic model. It features a linguistic encoder with a mixture alignment structure that incorporates pitch and energy predictors. This encoder processes phoneme sequences into hidden sequences and then a transformer decoder converts these into a mel-spectrogram. A post-net further refines this mel-spectrogram to improve reconstruction quality. This stage's output is a "coarse" mel-spectrogram that serves as prior knowledge for the next stage.
- Stage 2: Diffusion Decoder: This stage utilizes a conditional Generative Adversarial Network (GAN) for the denoising distribution. This GAN-based approach is key to accelerating the diffusion process. The diffusion decoder takes the coarse mel-spectrogram from Stage 1 and, through a shallow diffusion process, generates a high-fidelity mel-spectrogram. The "active shallow diffusion" aspect means the number of denoising steps is drastically reduced, in this case, to just one.
The model also incorporates a discriminator to ensure convergence between the actual and predicted denoising distributions.
Datasets
The CMG-TTS model was evaluated on two widely-used TTS datasets:
- AISHELL3: A large Mandarin Chinese dataset with 88,035 audio segments from 218 speakers, totaling 85 hours. This dataset is valuable for evaluating performance in tonal languages.
- LJSpeech: A standard English dataset containing 13,100 audio clips from a single speaker, with a total duration of 24 hours. This is commonly used for non-tonal language evaluation.
For training, a stratified random sampling approach was used, with 87,011 samples from AISHELL3 and 12,076 from LJSpeech. Validation and test sets consisted of 512 samples each.
Key Insights and Ideas
- Two-stage training is crucial for efficiency: By separating the acoustic modeling and diffusion processes into two stages, the model can learn valuable prior knowledge in the first stage, significantly simplifying the task for the diffusion model in the second stage.
- GANs accelerate diffusion: Replacing the traditional Gaussian function for denoising distribution with a GAN structure is a key enabler for faster inference in diffusion models.
- "Active shallow diffusion" is more than just fewer steps: It implies a more intelligent use of the diffusion process, where prior knowledge from a pre-trained model guides the limited diffusion steps, making them more effective.
- Linguistic features enhance prosody: The inclusion of pitch and energy predictors within the linguistic encoder directly contributes to better prosodic control and more natural-sounding speech.
- Post-net refines mel-spectrograms: The post-net plays a vital role in optimizing the reconstruction of the mel-spectrogram, leading to higher audio quality.
Key Results
- High-Quality Audio with Single Denoising Step: CMG-TTS achieves satisfactory subjective (MOS) and objective (MCD, F0 RMSE, PESQ, STOI, SegSNR) evaluation metrics with only one denoising step.
- Leading Real-Time Factor (RTF): Compared to other diffusion-based TTS models, CMG-TTS demonstrates a leading score in RTF, indicating significantly faster inference times. For example, on the AISHELL3 dataset, it achieved an RTF of 0.121, outperforming others.
- Ablation Studies Validate Components: Removing the post-net led to a significant degradation in audio quality, and removing both the post-net and the active shallow diffusion structure also resulted in a noticeable drop, confirming the effectiveness of these components.
- Competitive Performance: On AISHELL3, CMG-TTS achieved a MOS of 4.03 ± 0.07, and on LJSpeech, 4.08 ± 0.06, outperforming several other state-of-the-art models in terms of audio quality.
Limitations and Future Work
- Not Fully End-to-End: The paper acknowledges that CMG-TTS is not a fully end-to-end model because it still requires a separate vocoder (HiFi-GAN in this case) to synthesize the final audio. This necessitates separate training for the vocoder, adding training costs and potential error accumulation.
- Future Work: The authors suggest that future research should focus on developing fully end-to-end training mechanisms that can achieve comparable performance without sacrificing inference speed or audio quality. This would involve integrating the vocoder more seamlessly into the TTS pipeline.
Summary updated