WaveFlow: A Compact Flow-based Model for Raw Audio

Wei Ping, Kainan Peng, Kexin Zhao, Zhao Song

Abstract In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.

1. Introduction

Generating high-fidelity raw audio is a computationally demanding task that has historically forced a choice between quality and efficiency. Autoregressive (AR) models, such as WaveNet, produce high-quality audio but are slow due to their sequential nature. Flow-based models, like WaveGlow, allow for parallel synthesis but often require a massive number of parameters to match AR quality. Additionally, a "likelihood gap" persists where flow-based models struggle to fit the data distribution as effectively as AR models.

WaveFlow addresses these challenges by introducing a compact architecture that bridges the gap between these two paradigms. By reshaping 1-D audio into a 2-D matrix and applying local autoregression, WaveFlow achieves state-of-the-art quality with a significantly reduced parameter count and high inference speed.

2. Methods

WaveFlow's architecture is built around the concept of balancing parallelism with the expressivity of autoregressive modeling.

2.1 The 2-D Squeeze and Local Autoregression

The model employs a "2-D squeeze" operation, reshaping the 1-D waveform into a 2-D matrix with $h$ rows and $w$ columns. Unlike standard squeezing operations that can destroy temporal structure, this method preserves the sequence while enabling a novel dependency structure. WaveFlow is designed to be autoregressive specifically over the height ($h$) dimension.

The 2-D squeeze and receptive field patterns in WaveFlow. (a) illustrates the mapping from 1-D to 2-D. (b) and (c) show how the model captures dependencies across rows and columns using dilated 2-D convolutions.

As shown in Figure 2, the model uses a stack of 2-D dilated convolutions to capture long-range dependencies across the width (time) dimension while modeling local variations across the height dimension. This approach allows the model to generate waveforms in only $h$ sequential steps, where $h$ is typically small (e.g., 8 or 16), providing a massive speedup over fully sequential models.

2.2 A Unified Framework for 1-D Generative Models

WaveFlow provides a unified framework that generalizes existing likelihood-based models. By adjusting the height $h$, the model can transition between different architectural extremes.

Comparison of dependency structures in generative models. (a) represents a fully autoregressive model like WaveNet, and (b) represents a bipartite flow like WaveGlow. WaveFlow generalizes these by using a block-autoregressive structure.

When $h$ equals the full length of the audio, the model behaves as a standard AR model (Figure 1a). When $h=1$, it functions as a bipartite flow similar to WaveGlow (Figure 1b). WaveFlow leverages this flexibility to achieve the high log-likelihood of AR models while maintaining the synthesis efficiency of flow-based models.

3. Experimental Results

The model was evaluated on the LJ Speech dataset (22.05 kHz) and a proprietary 24 kHz female speech dataset.

3.1 Efficiency and Compactness

WaveFlow demonstrates remarkable efficiency. The small-footprint version uses only 5.91M parameters, making it 15$\times$ smaller than WaveGlow (87.88M parameters). Despite its small size, it synthesizes 22.05 kHz audio 42.6$\times$ faster than real-time on an NVIDIA V100 GPU without specialized inference kernels.

3.2 Audio Quality and Likelihood

In Mean Opinion Score (MOS) evaluations, WaveFlow achieved a score of 4.32, comparable to WaveGlow (4.34) and approaching WaveNet (4.43). Furthermore, WaveFlow effectively closed the likelihood gap, achieving a test log-likelihood of 5.055, nearly identical to WaveNet's 5.059. This indicates that the architecture is highly effective at capturing the true distribution of human speech.

4. Conclusion

WaveFlow offers a compact and efficient solution for raw audio generation. By treating audio as a 2-D signal and utilizing local autoregression, it combines the high fidelity of autoregressive models with the synthesis speed of flow-based models. Its small footprint and high performance make it a strong candidate for real-world speech synthesis applications and a potential framework for other sequential data domains.

Summary updated

Abstract

In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.
BibTeX
@inproceedings{ping2020waveflow,
  author = {Wei Ping and Kainan Peng and Kexin Zhao and Zhao Song},
  title = {{WaveFlow: A Compact Flow-based Model for Raw Audio}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=HFhlEpLAGlL},
}
Figure