Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram

Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim

TL;DR

Parallel WaveGAN is a "distillation-free" neural vocoder that generates high-quality speech waveforms significantly faster than real-time. By combining a non-autoregressive GAN architecture with a novel multi-resolution STFT loss, it eliminates the need for complex teacher-student training pipelines while maintaining competitive audio quality and a tiny memory footprint.

What Problem Does This Paper Address?

In the world of Text-to-Speech (TTS), autoregressive models like the original WaveNet set the gold standard for audio quality. However, because they generate audio one sample at a time, they are notoriously slow—often much slower than real-time.

To fix this, researchers previously developed "Parallel" versions (like Parallel WaveNet and ClariNet) using a technique called Probability Density Distillation. While fast, these models are a nightmare to train: they require a pre-trained "teacher" model and a complex, multi-stage optimization process that is prone to instability. The authors of this paper sought a way to get the speed of parallel generation without the headache of distillation.

Methods

The authors propose Parallel WaveGAN (PWG), which treats waveform generation as a Generative Adversarial Network (GAN) task.

  1. The Generator: A non-autoregressive WaveNet-style architecture. Unlike the original WaveNet, it uses non-causal convolutions and takes random Gaussian noise as input (conditioned on mel-spectrograms) to produce the entire waveform in one parallel pass.
  2. The Discriminator: A simple 1D convolutional network that tries to distinguish between real speech and the generator's "fake" speech.
  3. Multi-resolution STFT Loss: This is the "secret sauce." Because GANs can be unstable and might miss the fine-grained spectral details of speech, the authors add an auxiliary loss. They compute the Short-Time Fourier Transform (STFT) at three different scales (varying the FFT size, window length, and hop size). This forces the generator to capture the time-frequency distribution of speech accurately across different resolutions, preventing common artifacts like "metallic" noise.

Datasets

The model was trained and evaluated on a phonetically and prosaically balanced Japanese speech corpus featuring a professional female speaker.

  • Total duration: ~23 hours (11,449 utterances).
  • Sampling rate: 24 kHz.
  • Input features: 80-band log-mel spectrograms.

Key Insights and Ideas

  • Goodbye, Teacher: The most significant insight is that a well-designed auxiliary loss (the multi-resolution STFT) can replace the need for a teacher model entirely. This simplifies the TTS pipeline from a multi-week, multi-stage process to a single, end-to-end training session.
  • Resolution Trade-offs: In signal processing, there is a trade-off between time and frequency resolution. By using multiple STFT losses simultaneously, the model learns to reconstruct both sharp transients (like the "p" in "pop") and steady harmonics (like vowels) effectively.
  • Efficiency is King: With only 1.44 million parameters, PWG is remarkably lightweight. This makes it a prime candidate for deployment on mobile devices or servers where memory and compute are at a premium.

Key Results

  • Inference Speed: PWG generates 24 kHz audio 28.68 times faster than real-time on a single NVIDIA V100 GPU. This is nearly twice as fast as the distillation-based ClariNet model.
  • Training Speed: The training process is 4.82x faster than ClariNet-GAN (2.8 days vs. 13.5 days), primarily because it skips the teacher-training and distillation phases.
  • Audio Quality: In Mean Opinion Score (MOS) tests, PWG achieved a score of 4.16, which is statistically competitive with the best distillation-based models (ClariNet-GAN at 4.14) and significantly better than the original autoregressive WaveNet (3.61) when used in a full TTS pipeline.

Limitations and Future Work

  • Phase Information: The current multi-resolution STFT loss focuses on the magnitude of the spectrum. The authors suggest that explicitly incorporating phase-related losses could further improve audio clarity.
  • Generalization: While the results on the Japanese corpus are excellent, the authors note that the model needs to be tested on more diverse and "expressive" datasets (e.g., audio with high emotional variance) to ensure it remains robust.
  • Discriminator Sensitivity: While the generator is small, the training still relies on a discriminator. Future work could explore if the discriminator architecture itself can be further optimized to improve the generator's performance.

Summary updated

Abstract

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.
BibTeX
@inproceedings{yamamoto2020parallel,
  author = {Ryuichi Yamamoto and Eunwoo Song and Jae-Min Kim},
  title = {{Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=uxhU5H2bbV},
}

Loading related content...

Figure