Probability Density Distillation with Generative Adversarial Networks for High-Quality Parallel Waveform Generation

Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim

Abstract

This paper proposes an effective probability density distillation (PDD) algorithm for WaveNet-based parallel waveform generation (PWG) systems. Recently proposed teacher-student frameworks in the PWG system have successfully achieved a real-time generation of speech signals. However, the difficulties optimizing the PDD criteria without auxiliary losses result in quality degradation of synthesized speech. To generate more natural speech signals within the teacher-student framework, we propose a novel optimization criterion based on generative adversarial networks (GANs). In the proposed method, the inverse autoregressive flow-based student model is incorporated as a generator in the GAN framework, and jointly optimized by the PDD mechanism with the proposed adversarial learning method. As this process encourages the student to model the distribution of realistic speech waveform, the perceptual quality of the synthesized speech becomes much more natural. Our experimental results verify that the PWG systems with the proposed method outperform both those using conventional approaches, and also autoregressive generation systems with a well-trained teacher WaveNet.

Introduction

Neural vocoders, such as WaveNet, have set the standard for high-fidelity speech synthesis by converting acoustic features into raw waveforms. However, WaveNet's autoregressive nature—generating audio one sample at a time—limits its utility in real-time applications. To overcome this, Parallel WaveNet and ClariNet introduced a teacher-student framework using Probability Density Distillation (PDD). In this setup, a pre-trained teacher WaveNet distills its knowledge into a parallel student model (Inverse Autoregressive Flow, or IAF) using Kullback-Leibler Divergence (KLD).

Despite its efficiency, KLD-based distillation often hits a "quality ceiling," where the student cannot surpass the teacher. Furthermore, relying solely on KLD or basic spectral losses can lead to artifacts like metallic noise or "whispering" because these metrics do not fully capture the complex high-frequency distributions of natural speech.

Proposed Method: PDD-GAN

The authors propose a joint optimization strategy that integrates Generative Adversarial Networks (GANs) into the distillation process. The student model acts as a generator, while a separate 10-layer non-causal dilated 1-D CNN serves as a discriminator.

Schematic diagram of the proposed PDD-GAN framework, showing the joint optimization of KLD, auxiliary, and adversarial losses.

The optimization utilizes three primary loss functions:

  1. KLD Loss ($L_{KLD}$): Forces the student to mimic the teacher's probability distribution.
  2. STFT-based Auxiliary Loss ($L_{AUX}$): Compares the Short-Time Fourier Transform of generated audio against ground truth to capture spectral characteristics.
  3. Adversarial Loss ($L_{ADV}$): Uses a Least-Squares GAN (LSGAN) approach to force the student to produce waveforms indistinguishable from real speech.

A critical finding was the importance of a "warm-up" period using KLD and STFT losses before introducing adversarial training to ensure stability.

Experimental Setup and Results

The method was evaluated on a Japanese speech corpus (approx. 7.34 hours of training data) from a professional female speaker. The input features were 80-band log-mel spectrograms, and the audio was sampled at 24 kHz.

Subjective evaluation via Mean Opinion Score (MOS) tests with 14 native speakers demonstrated the effectiveness of the proposed approach.

MOS scores comparing the raw audio, teacher WaveNet, and various student configurations. The proposed KLAXAD model outperforms the teacher.

Key findings include:

  • Surpassing the Teacher: The proposed KLAXAD model achieved a MOS of 4.19, significantly higher than the teacher WaveNet's 4.11 and the standard distillation model (KLAX) at 4.06.
  • The Role of KLD: Models trained without KLD (AXAD) performed poorly, indicating that KLD is essential for guiding the student toward the correct distribution before GAN fine-tuning.
  • Weight Refinement: The authors found that reducing the KLD loss weight late in training ($KLAXAD^*$) further improved naturalness, as the teacher's distribution can eventually become a constraint rather than a guide.

Conclusion

By combining PDD with adversarial learning, this research successfully breaks the quality ceiling of traditional teacher-student distillation for speech synthesis. The resulting parallel models generate high-quality speech that exceeds the naturalness of the original autoregressive teacher. Future work includes exploring learnable loss weights and validating the approach on multi-speaker and multi-lingual datasets.

Summary updated

Abstract

This paper proposes an effective probability density distillation (PDD) algorithm for WaveNet-based parallel waveform generation (PWG) systems. Recently proposed teacher-student frameworks in the PWG system have successfully achieved a real-time generation of speech signals. However, the difficulties optimizing the PDD criteria without auxiliary losses result in quality degradation of synthesized speech. To generate more natural speech signals within the teacher-student framework, we propose a novel optimization criterion based on generative adversarial networks (GANs). In the proposed method, the inverse autoregressive flow-based student model is incorporated as a generator in the GAN framework, and jointly optimized by the PDD mechanism with the proposed adversarial learning method. As this process encourages the student to model the distribution of realistic speech waveform, the perceptual quality of the synthesized speech becomes much more natural. Our experimental results verify that the PWG systems with the proposed method outperform both those using conventional approaches, and also autoregressive generation systems with a well-trained teacher WaveNet.
BibTeX
@inproceedings{yamamoto2019probability,
  author = {Ryuichi Yamamoto and Eunwoo Song and Jae-Min Kim},
  title = {{Probability Density Distillation with Generative Adversarial Networks for High-Quality Parallel Waveform Generation}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=h8RFyPSStt},
}
Figure