WaveFlow: A Compact Flow-based Model for Raw Audio

Wei Ping, Kainan Peng, Kexin Zhao, Zhao Song

TL;DR

WaveFlow is a compact, flow-based generative model for raw audio that bridges the gap between high-fidelity but slow autoregressive models (like WaveNet) and fast but massive flow models (like WaveGlow). By using a novel 2-D dilated convolutional architecture that is autoregressive over a "squeezed" height dimension, it achieves state-of-the-art speech quality with a 15x smaller footprint than WaveGlow and synthesis speeds over 40x faster than real-time.

The Challenge of Raw Audio Generation

Generating high-fidelity raw audio is a major challenge in deep learning. Existing solutions generally fall into three camps, each with a significant drawback:

  1. Autoregressive (AR) models (e.g., WaveNet): These provide the highest audio quality and likelihood scores but are prohibitively slow because they generate audio sequentially, sample-by-sample.
  2. Bipartite Flow models (e.g., WaveGlow): These allow for fast parallel synthesis but require an enormous number of parameters (often 80M+) to match the quality of AR models. They also tend to lose temporal information by "squeezing" audio into channels too aggressively.
  3. Distillation-based models (e.g., Parallel WaveNet): These are fast but require a complex training pipeline involving a pre-trained "teacher" model and multiple auxiliary loss functions.

The authors of WaveFlow sought a model that is easy to train (using simple maximum likelihood), fast to sample, high-fidelity, and has a small memory footprint suitable for on-device deployment.

The WaveFlow Architecture

WaveFlow introduces a unified framework that treats WaveNet and WaveGlow as special cases. Its core innovation lies in how it structures the flow transformation:

  • 2-D Squeezing: Instead of squeezing 1-D audio into many channels, WaveFlow reshapes the 1-D waveform into an $h \times w$ 2-D matrix.
  • Local Autoregression: The model is designed to be autoregressive only over the height dimension ($h$). This means to generate a full matrix, it only needs $h$ sequential steps. Since $h$ is typically small (e.g., 8 or 16), this is orders of magnitude faster than the thousands of steps required by standard AR models.
  • 2-D Dilated Convolutions: It uses a stack of 2-D dilated convolutions to handle long-range dependencies in the audio while modeling local variations expressively.
  • Unified View: By adjusting the height $h$, WaveFlow can trade off between model capacity and synthesis parallelism. If $h=1$, it behaves like a bipartite flow; if $h=n$ (the full length), it becomes a standard AR model.
Jacobian matrices for (a) an autoregressive transformation and (b) a bipartite transformation. WaveFlow bridges these extremes.

Key Insights and Receptive Fields

  • The "Likelihood Gap": The paper demonstrates that the performance gap between AR and flow models is largely due to the limited expressiveness of bipartite transformations. By introducing a small amount of sequentiality (the height $h$), WaveFlow significantly closes this gap.
  • Receptive Field Matters: The authors found that for good results, the receptive field over the height dimension must be at least as large as the height $h$ itself. This ensures the model doesn't introduce unnecessary conditional independence.
  • Bidirectional Modeling via Permutation: Stacking multiple flows with different permutation strategies (like reversing the height order) allows the model to look at both "past" and "future" samples, which is crucial for high-fidelity synthesis.
Comparison of receptive fields for (a) WaveFlow, (b) WaveGlow, and (c) WaveNet. WaveFlow's 2-D structure allows for a flexible and efficient receptive field.

Key Results

The researchers evaluated WaveFlow on two main datasets: LJ Speech (for primary likelihood and quality benchmarks) and a proprietary TTS dataset (to test WaveFlow as a vocoder within a full text-to-speech pipeline).

  • Compactness: A WaveFlow model with only 5.91M parameters achieved comparable audio quality to a WaveGlow model with 268M parameters.
  • Speed: It synthesizes 22.05 kHz audio 42.6x faster than real-time on a single V100 GPU, without needing specialized, highly-engineered inference kernels.
  • Quality: In Mean Opinion Score (MOS) tests, WaveFlow scored 4.32, very close to the ground truth (4.56) and standard WaveNet (4.43), and significantly higher than smaller versions of WaveGlow.
  • Likelihood: WaveFlow achieved higher test log-likelihoods than WaveGlow, confirming it captures the data distribution more accurately.

Limitations and Future Work

While WaveFlow is much faster than WaveNet, it is not fully parallel like WaveGlow; it still requires $h$ sequential steps. Although $h$ is small, this could be a bottleneck for extremely low-latency requirements.

The authors suggest that future work could involve exploring different 2-D architectures or more sophisticated permutation strategies to further reduce the number of sequential steps while maintaining the high likelihood scores that make WaveFlow so efficient.

Summary updated

Abstract

In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dil...
BibTeX
@inproceedings{ping2020waveflow,
  author = {Wei Ping and Kainan Peng and Kexin Zhao and Zhao Song},
  title = {{WaveFlow: A Compact Flow-based Model for Raw Audio}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=HFhlEpLAGlL},
}
Figure