Abstract
We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.
Introduction
In the field of Text-to-Speech (TTS), autoregressive models like the original WaveNet have long been the standard for high-quality audio. However, their sequential generation process is computationally intensive and slow. While subsequent "parallel" models like Parallel WaveNet and ClariNet achieved real-time speeds, they relied on Probability Density Distillation. This technique requires a complex teacher-student training pipeline that is often unstable and difficult to optimize.
Parallel WaveGAN (PWG) addresses these issues by introducing a "distillation-free" neural vocoder. By treating waveform generation as a Generative Adversarial Network (GAN) task and incorporating a novel multi-resolution STFT loss, PWG achieves high-speed generation and high-fidelity audio without the overhead of a teacher model.
Methodology
The Parallel WaveGAN framework consists of a generator and a discriminator optimized through a combination of adversarial and auxiliary losses.
Architecture
- The Generator: A non-autoregressive WaveNet-style architecture using non-causal convolutions. It takes random Gaussian noise as input, conditioned on mel-spectrograms, to produce the entire waveform in a single parallel pass.
- The Discriminator: A simple 1D convolutional network that distinguishes between real speech and the generator's synthetic output.
Multi-resolution STFT Loss
The "secret sauce" of PWG is the multi-resolution Short-Time Fourier Transform (STFT) loss. Because GANs can be unstable and may miss fine-grained spectral details, the authors apply an auxiliary loss computed at three different scales (varying FFT size, window length, and hop size). This forces the generator to capture the time-frequency distribution of speech accurately across different resolutions, preventing artifacts like "metallic" noise and ensuring the reconstruction of both sharp transients and steady harmonics.
Experimental Setup
The model was trained and evaluated on a phonetically and prosodically balanced Japanese speech corpus featuring a professional female speaker. The dataset included approximately 23 hours of audio (11,449 utterances) sampled at 24 kHz. The input features were 80-band log-mel spectrograms.
Results and Discussion
Parallel WaveGAN demonstrates significant advantages in efficiency, training speed, and audio quality.
Efficiency and Speed
With only 1.44 million parameters, PWG is remarkably lightweight. It generates 24 kHz audio 28.68 times faster than real-time on a single NVIDIA V100 GPU, nearly doubling the speed of the distillation-based ClariNet model. Additionally, the training process is 4.82x faster than ClariNet-GAN (2.8 days vs. 13.5 days) because it eliminates the teacher-training and distillation phases.
Audio Quality
Perceptual listening tests (MOS) confirm that PWG is highly competitive. As shown in the comparison below, the multi-resolution STFT loss is critical; without it (System 5), quality drops significantly. With the full loss suite (System 6), PWG achieves a MOS of 4.06, which rises to 4.16 when integrated into a full Transformer-based TTS pipeline—statistically comparable to the best distillation-based systems.
Conclusion
Parallel WaveGAN offers a streamlined and efficient solution for high-quality speech synthesis. By replacing complex distillation with a multi-resolution STFT loss, it simplifies the TTS pipeline while maintaining state-of-the-art performance and a tiny memory footprint. Future work will explore incorporating phase-related losses and testing the model's robustness on more diverse and expressive datasets.
Summary updated