Cauchy Diffusion: A Heavy-tailed Denoising Diffusion Probabilistic Model for Speech Synthesis

Qi Lian, Yu Qi, Yueming WangAAAI 2025 · DOI

Introduction

Standard Denoising Diffusion Probabilistic Models (DDPMs) have become a cornerstone of high-fidelity speech synthesis, particularly in the role of vocoders that transform mel-spectrograms into raw audio. However, these models often suffer from "mode collapse," where the generated speech lacks prosodic variety and sounds robotic. This paper introduces Cauchy Diffusion, a novel framework that replaces traditional thin-tailed Gaussian noise with heavy-tailed Cauchy noise to better capture the diverse intonations and rhythms inherent in human speech.

The Challenge: Mode Collapse in Prosody

In speech synthesis, "prosody" refers to the complex patterns of stress, intonation, and rhythm. Because a single sentence can be spoken in many valid ways, the underlying data distribution is highly imbalanced. Standard DDPMs rely on Gaussian distributions, which concentrate probability mass near the mean. This "thin-tailed" property causes models to ignore rare prosodic variations (outliers), leading them to converge on a "safe" but monotonous average. This results in synthesized voices that lack natural character and emotional range.

Methodology: The Cauchy Distribution

To address this, the authors propose using the Cauchy distribution. Unlike the Gaussian distribution, the Cauchy distribution is "heavy-tailed," meaning it assigns significantly higher probability to extreme values. This allows the model to remain resilient to the diverse "noise" of varied prosodic expressions during training.

Implementing a Cauchy-based diffusion process is mathematically challenging because the Cauchy distribution does not satisfy the standard Kolmogorov equations used for diffusion schedules. The authors overcome this using the Ratio Distribution Trick, representing the Cauchy distribution as the ratio of two independent Gaussian distributions. This bridge allows the derivation of a stable diffusion schedule while maintaining the heavy-tailed benefits.

Overview of the Gaussian and Cauchy diffusion processes, illustrating the ratio distribution trick used to derive the Cauchy schedule.

Key technical contributions include:

  • Stable Diffusion Schedules: Derived via the ratio of two Gaussian processes.
  • Tailored Optimization: A specific KL-divergence loss function designed for Cauchy noise scales.
  • Stochastic Sampling: A sampling methodology that introduces the randomness necessary to generate diverse speech outputs from identical inputs.

Experimental Results

The model was evaluated on the LJSpeech (single-speaker) and VCTK (multi-speaker) datasets against state-of-the-art baselines like DiffWave, WaveGrad, and HiFiGAN.

Audio Quality and Diversity

Cauchy Diffusion demonstrated superior performance across both objective and subjective metrics. On the LJSpeech dataset, it achieved a PESQ score of 4.055, outperforming all Gaussian-based diffusion models and GAN-based vocoders. Crucially, it significantly improved prosodic diversity, as measured by Mel Cepstral Distortion (MCD). Cauchy Diffusion reached an MCD of 1.862 on LJSpeech, compared to 2.062 for DiffWave, indicating a much closer match to the natural variation of human speech.

Comparison of Cauchy Diffusion against various baselines on the LJSpeech dataset, showing state-of-the-art performance in PESQ, STOI, MCD, and MOS.

Human Evaluation

In Mean Opinion Score (MOS) tests, human listeners rated Cauchy Diffusion at 4.00, effectively matching the ground truth audio (3.94) within the margin of error. This confirms that the mathematical improvements in diversity translate directly to a more natural listening experience.

Discussion and Limitations

While Cauchy Diffusion sets a new standard for speech naturalness and diversity, it inherits the primary drawback of diffusion models: inference speed. The iterative nature of the denoising process makes it slower than "one-shot" GAN models like HiFiGAN. Although the model remains effective at 50 steps, it is not yet optimized for the real-time, low-latency requirements of some production environments.

Conclusion

By swapping Gaussian noise for heavy-tailed Cauchy noise, Cauchy Diffusion provides a robust solution to mode collapse in speech synthesis. The use of the ratio distribution trick provides a mathematical blueprint for integrating other non-Gaussian distributions into generative frameworks, opening new doors for modeling complex, imbalanced data across various domains.

Summary updated

BibTeX
@inproceedings{lian2025cauchy,
  author = {Qi Lian and Yu Qi and Yueming Wang},
  title = {{Cauchy Diffusion: A Heavy-tailed Denoising Diffusion Probabilistic Model for Speech Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i23.34634},
}

Loading related content...

Figure