Abstract
High-quality speech synthesis typically relies on autoregressive models like WaveNet, which are computationally expensive for real-time applications. While teacher-student distillation frameworks allow for parallel generation, they often suffer from a quality ceiling imposed by the teacher. This paper proposes a novel method combining Probability Density Distillation (PDD) with Generative Adversarial Networks (GANs). By incorporating an adversarial loss alongside distillation and auxiliary spectral losses, the student model—an Inverse Autoregressive Flow (IAF)—is able to surpass the audio quality of its teacher.
Introduction
High-quality speech synthesis has long been dominated by autoregressive models such as WaveNet. While these models produce high-fidelity audio, their sequential nature makes them slow, often requiring more time to generate audio than the duration of the audio itself. To address this, researchers developed "teacher-student" frameworks like Parallel WaveNet and ClariNet. In these setups, a slow, well-trained teacher (WaveNet) distills its knowledge into a fast, parallel student model based on Inverse Autoregressive Flow (IAF).
However, these systems face two significant hurdles:
- The Quality Ceiling: A student trained solely to mimic a teacher typically cannot exceed the teacher's performance.
- Optimization Difficulties: Relying only on divergence-based losses (like KLD) often results in "whispering" artifacts or noisy audio as the model struggles to capture fine-grained speech distributions.
Proposed Method
The authors propose a unified optimization criterion that combines three distinct types of losses to train the IAF student model:
- KLD Distillation Loss: The standard loss where the student matches the probability distribution of a pre-trained WaveNet teacher.
- STFT-based Auxiliary Loss: A loss comparing frequency-domain characteristics (spectrograms) of generated audio against ground-truth recordings to improve texture and reduce artifacts.
- Adversarial Loss (GAN): The core novelty. The student model acts as a Generator, and a new Discriminator network is introduced to distinguish between real speech and generated speech. This forces the student to produce realistic waveforms that capture nuances from the ground truth.
The student model is an IAF-based neural vocoder for parallel sampling, while the Discriminator is a non-causal dilated 1-D convolutional network designed to capture sample-level details.
Experimental Setup
The researchers utilized a phonetically and prosaically balanced Japanese speech corpus featuring a single female professional speaker. The dataset consisted of approximately 7.34 hours (3,299 utterances) for training, with 0.89 hours for development and 0.92 hours for evaluation. Audio was sampled at 24 kHz with 16-bit quantization.
Key Results
The researchers conducted Mean Opinion Score (MOS) tests where native speakers rated the audio quality on a scale of 1 to 5.
The results demonstrate that the proposed method (KLAXAD) achieved a MOS of 4.19, surpassing both the conventional distillation approach (KLAX, 4.06) and the original autoregressive Teacher WaveNet (4.11). This confirms that by including a GAN loss against real data, the student is no longer strictly bound by the teacher's limitations and can learn nuances from the ground truth that the teacher might have missed.
Conclusion
This work successfully demonstrates that combining probability density distillation with adversarial training can break the "teacher ceiling" in parallel waveform generation. A critical finding was the necessity of a "warm-up" period where the student is trained only with KLD and STFT losses before introducing adversarial training. Future research will explore making the loss weight coefficients ($\lambda_{kld}, \lambda_{aux}, \lambda_{adv}$) learnable to further optimize performance and reduce manual hyperparameter tuning.
Summary updated