Probability density distillation with generative adversarial networks for high-quality parallel waveform generation

Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim

Abstract

High-quality speech synthesis typically relies on autoregressive models like WaveNet, which are computationally expensive for real-time applications. While teacher-student distillation frameworks allow for parallel generation, they often suffer from a quality ceiling imposed by the teacher. This paper proposes a novel method combining Probability Density Distillation (PDD) with Generative Adversarial Networks (GANs). By incorporating an adversarial loss alongside distillation and auxiliary spectral losses, the student model—an Inverse Autoregressive Flow (IAF)—is able to surpass the audio quality of its teacher.

Introduction

High-quality speech synthesis has long been dominated by autoregressive models such as WaveNet. While these models produce high-fidelity audio, their sequential nature makes them slow, often requiring more time to generate audio than the duration of the audio itself. To address this, researchers developed "teacher-student" frameworks like Parallel WaveNet and ClariNet. In these setups, a slow, well-trained teacher (WaveNet) distills its knowledge into a fast, parallel student model based on Inverse Autoregressive Flow (IAF).

However, these systems face two significant hurdles:

  1. The Quality Ceiling: A student trained solely to mimic a teacher typically cannot exceed the teacher's performance.
  2. Optimization Difficulties: Relying only on divergence-based losses (like KLD) often results in "whispering" artifacts or noisy audio as the model struggles to capture fine-grained speech distributions.

Proposed Method

The authors propose a unified optimization criterion that combines three distinct types of losses to train the IAF student model:

  1. KLD Distillation Loss: The standard loss where the student matches the probability distribution of a pre-trained WaveNet teacher.
  2. STFT-based Auxiliary Loss: A loss comparing frequency-domain characteristics (spectrograms) of generated audio against ground-truth recordings to improve texture and reduce artifacts.
  3. Adversarial Loss (GAN): The core novelty. The student model acts as a Generator, and a new Discriminator network is introduced to distinguish between real speech and generated speech. This forces the student to produce realistic waveforms that capture nuances from the ground truth.
Overview of the proposed training framework compared to conventional distillation methods.

The student model is an IAF-based neural vocoder for parallel sampling, while the Discriminator is a non-causal dilated 1-D convolutional network designed to capture sample-level details.

Experimental Setup

The researchers utilized a phonetically and prosaically balanced Japanese speech corpus featuring a single female professional speaker. The dataset consisted of approximately 7.34 hours (3,299 utterances) for training, with 0.89 hours for development and 0.92 hours for evaluation. Audio was sampled at 24 kHz with 16-bit quantization.

Key Results

The researchers conducted Mean Opinion Score (MOS) tests where native speakers rated the audio quality on a scale of 1 to 5.

Mean Opinion Score (MOS) results comparing the proposed KLAXAD method against the teacher and other baselines.

The results demonstrate that the proposed method (KLAXAD) achieved a MOS of 4.19, surpassing both the conventional distillation approach (KLAX, 4.06) and the original autoregressive Teacher WaveNet (4.11). This confirms that by including a GAN loss against real data, the student is no longer strictly bound by the teacher's limitations and can learn nuances from the ground truth that the teacher might have missed.

Conclusion

This work successfully demonstrates that combining probability density distillation with adversarial training can break the "teacher ceiling" in parallel waveform generation. A critical finding was the necessity of a "warm-up" period where the student is trained only with KLD and STFT losses before introducing adversarial training. Future research will explore making the loss weight coefficients ($\lambda_{kld}, \lambda_{aux}, \lambda_{adv}$) learnable to further optimize performance and reduce manual hyperparameter tuning.

Summary updated

BibTeX
@inproceedings{yamamoto2019probability,
  author = {Ryuichi Yamamoto and Eunwoo Song and Jae-Min Kim},
  title = {{Probability density distillation with generative adversarial networks for high-quality parallel waveform generation}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=h8RFyPSStt},
}
Figure