Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram

Ryuichi Yamamoto, Eunwoo Song, Jae-Min Kim

Abstract

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.

Introduction

In the field of Text-to-Speech (TTS), autoregressive models like the original WaveNet have long been the standard for high-quality audio. However, their sequential generation process is computationally intensive and slow. While subsequent "parallel" models like Parallel WaveNet and ClariNet achieved real-time speeds, they relied on Probability Density Distillation. This technique requires a complex teacher-student training pipeline that is often unstable and difficult to optimize.

Parallel WaveGAN (PWG) addresses these issues by introducing a "distillation-free" neural vocoder. By treating waveform generation as a Generative Adversarial Network (GAN) task and incorporating a novel multi-resolution STFT loss, PWG achieves high-speed generation and high-fidelity audio without the overhead of a teacher model.

Methodology

The Parallel WaveGAN framework consists of a generator and a discriminator optimized through a combination of adversarial and auxiliary losses.

Architecture

  1. The Generator: A non-autoregressive WaveNet-style architecture using non-causal convolutions. It takes random Gaussian noise as input, conditioned on mel-spectrograms, to produce the entire waveform in a single parallel pass.
  2. The Discriminator: A simple 1D convolutional network that distinguishes between real speech and the generator's synthetic output.

Multi-resolution STFT Loss

The "secret sauce" of PWG is the multi-resolution Short-Time Fourier Transform (STFT) loss. Because GANs can be unstable and may miss fine-grained spectral details, the authors apply an auxiliary loss computed at three different scales (varying FFT size, window length, and hop size). This forces the generator to capture the time-frequency distribution of speech accurately across different resolutions, preventing artifacts like "metallic" noise and ensuring the reconstruction of both sharp transients and steady harmonics.

Architecture of Parallel WaveGAN showing the generator, discriminator, and multi-resolution STFT loss components.

Experimental Setup

The model was trained and evaluated on a phonetically and prosodically balanced Japanese speech corpus featuring a professional female speaker. The dataset included approximately 23 hours of audio (11,449 utterances) sampled at 24 kHz. The input features were 80-band log-mel spectrograms.

Results and Discussion

Parallel WaveGAN demonstrates significant advantages in efficiency, training speed, and audio quality.

Efficiency and Speed

With only 1.44 million parameters, PWG is remarkably lightweight. It generates 24 kHz audio 28.68 times faster than real-time on a single NVIDIA V100 GPU, nearly doubling the speed of the distillation-based ClariNet model. Additionally, the training process is 4.82x faster than ClariNet-GAN (2.8 days vs. 13.5 days) because it eliminates the teacher-training and distillation phases.

Audio Quality

Perceptual listening tests (MOS) confirm that PWG is highly competitive. As shown in the comparison below, the multi-resolution STFT loss is critical; without it (System 5), quality drops significantly. With the full loss suite (System 6), PWG achieves a MOS of 4.06, which rises to 4.16 when integrated into a full Transformer-based TTS pipeline—statistically comparable to the best distillation-based systems.

Comparison of various systems across model size, inference speed, and Mean Opinion Score (MOS).

Conclusion

Parallel WaveGAN offers a streamlined and efficient solution for high-quality speech synthesis. By replacing complex distillation with a multi-resolution STFT loss, it simplifies the TTS pipeline while maintaining state-of-the-art performance and a tiny memory footprint. Future work will explore incorporating phase-related losses and testing the model's robustness on more diverse and expressive datasets.

Summary updated

Abstract

We propose Parallel WaveGAN, a distillation-free, fast, and small-footprint waveform generation method using a generative adversarial network. In the proposed method, a non-autoregressive WaveNet is trained by jointly optimizing multi-resolution spectrogram and adversarial loss functions, which can effectively capture the time-frequency distribution of the realistic speech waveform. As our method does not require density distillation used in the conventional teacher-student framework, the entire model can be easily trained. Furthermore, our model is able to generate high-fidelity speech even with its compact architecture. In particular, the proposed Parallel WaveGAN has only 1.44 M parameters and can generate 24 kHz speech waveform 28.68 times faster than real-time on a single GPU environment. Perceptual listening test results verify that our proposed method achieves 4.16 mean opinion score within a Transformer-based text-to-speech framework, which is comparative to the best distillation-based Parallel WaveNet system.
BibTeX
@inproceedings{yamamoto2020parallel,
  author = {Ryuichi Yamamoto and Eunwoo Song and Jae-Min Kim},
  title = {{Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=uxhU5H2bbV},
}
Figure