High-Quality Text-to-Speech Implementation via Active Shallow Diffusion Mechanism

Junlin Deng, Ruihua Hou, Yan Deng, Yu Long, Ning WuSensors 2025 · DOI

TL;DR

This paper introduces CMG-TTS, a novel two-stage text-to-speech (TTS) model that significantly speeds up diffusion-based synthesis by employing an "active shallow diffusion mechanism." By training a basic acoustic model first to provide prior knowledge, CMG-TTS achieves high-quality speech with just one denoising step, outperforming existing diffusion TTS models in real-time factor (RTF).

What Problem Does This Paper Address?

Traditional diffusion models, while powerful for generative tasks like speech synthesis, suffer from slow inference times due to the need for hundreds of iterative denoising steps. This makes real-time TTS applications challenging. Existing diffusion-based TTS models have attempted to address this by reducing the number of denoising steps, but often at the cost of speech quality or by introducing complex alignment mechanisms. The paper aims to bridge this gap by developing a TTS model that is both high-quality and efficient for real-time inference, specifically by optimizing the diffusion process.

Methods

The core innovation is the Cascaded MixGAN-TTS (CMG-TTS), a two-stage architecture that leverages an active shallow diffusion mechanism.

  • Stage 1: Basic Acoustic Model: This stage trains a foundational acoustic model. It features a linguistic encoder with a mixture alignment structure that incorporates pitch and energy predictors. This encoder processes phoneme sequences into hidden sequences and then a transformer decoder converts these into a mel-spectrogram. A post-net further refines this mel-spectrogram to improve reconstruction quality. This stage's output is a "coarse" mel-spectrogram that serves as prior knowledge for the next stage.
  • Stage 2: Diffusion Decoder: This stage utilizes a conditional Generative Adversarial Network (GAN) for the denoising distribution. This GAN-based approach is key to accelerating the diffusion process. The diffusion decoder takes the coarse mel-spectrogram from Stage 1 and, through a shallow diffusion process, generates a high-fidelity mel-spectrogram. The "active shallow diffusion" aspect means the number of denoising steps is drastically reduced, in this case, to just one.

The model also incorporates a discriminator to ensure convergence between the actual and predicted denoising distributions.

Datasets

The CMG-TTS model was evaluated on two widely-used TTS datasets:

  • AISHELL3: A large Mandarin Chinese dataset with 88,035 audio segments from 218 speakers, totaling 85 hours. This dataset is valuable for evaluating performance in tonal languages.
  • LJSpeech: A standard English dataset containing 13,100 audio clips from a single speaker, with a total duration of 24 hours. This is commonly used for non-tonal language evaluation.

For training, a stratified random sampling approach was used, with 87,011 samples from AISHELL3 and 12,076 from LJSpeech. Validation and test sets consisted of 512 samples each.

Key Insights and Ideas

  • Two-stage training is crucial for efficiency: By separating the acoustic modeling and diffusion processes into two stages, the model can learn valuable prior knowledge in the first stage, significantly simplifying the task for the diffusion model in the second stage.
  • GANs accelerate diffusion: Replacing the traditional Gaussian function for denoising distribution with a GAN structure is a key enabler for faster inference in diffusion models.
  • "Active shallow diffusion" is more than just fewer steps: It implies a more intelligent use of the diffusion process, where prior knowledge from a pre-trained model guides the limited diffusion steps, making them more effective.
  • Linguistic features enhance prosody: The inclusion of pitch and energy predictors within the linguistic encoder directly contributes to better prosodic control and more natural-sounding speech.
  • Post-net refines mel-spectrograms: The post-net plays a vital role in optimizing the reconstruction of the mel-spectrogram, leading to higher audio quality.

Key Results

  • High-Quality Audio with Single Denoising Step: CMG-TTS achieves satisfactory subjective (MOS) and objective (MCD, F0 RMSE, PESQ, STOI, SegSNR) evaluation metrics with only one denoising step.
  • Leading Real-Time Factor (RTF): Compared to other diffusion-based TTS models, CMG-TTS demonstrates a leading score in RTF, indicating significantly faster inference times. For example, on the AISHELL3 dataset, it achieved an RTF of 0.121, outperforming others.
  • Ablation Studies Validate Components: Removing the post-net led to a significant degradation in audio quality, and removing both the post-net and the active shallow diffusion structure also resulted in a noticeable drop, confirming the effectiveness of these components.
  • Competitive Performance: On AISHELL3, CMG-TTS achieved a MOS of 4.03 ± 0.07, and on LJSpeech, 4.08 ± 0.06, outperforming several other state-of-the-art models in terms of audio quality.

Limitations and Future Work

  • Not Fully End-to-End: The paper acknowledges that CMG-TTS is not a fully end-to-end model because it still requires a separate vocoder (HiFi-GAN in this case) to synthesize the final audio. This necessitates separate training for the vocoder, adding training costs and potential error accumulation.
  • Future Work: The authors suggest that future research should focus on developing fully end-to-end training mechanisms that can achieve comparable performance without sacrificing inference speed or audio quality. This would involve integrating the vocoder more seamlessly into the TTS pipeline.

Summary updated

Abstract

Denoising diffusion probabilistic models (DDPMs) have proven to be useful in text-to-speech (TTS) tasks; however, it has been a challenge for traditional diffusion models to carry out real-time processing because of the need for hundreds of sampling steps during the iteration. In this work, a two-stage fast inference and efficient diffusion-based acoustic model of TTS, the Cascaded MixGAN-TTS (CMG-TTS), is proposed to address this problem. An active shallow diffusion mechanism is adopted to divide the CMG-TTS training process into two stages. Specifically, a basic acoustic model in the first stage is trained to provide valuable a priori knowledge for the second stage, and for the underlying acoustic modeling, a mixture combination mechanism-based linguistic encoder is introduced to work with pitch and energy predictors. In the following stage of processing, a post-net is used to optimize the mel-spectrogram reconstruction performance. The CMG-TTS is evaluated on datasets such as the AISHELL3 and LJSpeech, and the experiments show that the CMG-TTS achieves satisfactory results in both subjective and objective evaluation metrics with only one denoising step. Compared to other TTS models based on diffusion modeling, the CMG-TTS obtains a leading score in the real time factor (RTF), and both stages of the CMG-TTS are effective in the ablation studies.
BibTeX
@inproceedings{deng2025high,
  author = {Junlin Deng and Ruihua Hou and Yan Deng and Yu Long and Ning Wu},
  title = {{High-Quality Text-to-Speech Implementation via Active Shallow Diffusion Mechanism}},
  booktitle = {Sensors},
  year = {2025},
  doi = {10.3390/s25030833},
}
Figure