Abstract In this work, we propose WaveFlow, a small-footprint generative flow for raw audio, which is directly trained with maximum likelihood. It handles the long-range structure of 1-D waveform with a dilated 2-D convolutional architecture, while modeling the local variations using expressive autoregressive functions. WaveFlow provides a unified view of likelihood-based models for 1-D data, including WaveNet and WaveGlow as special cases. It generates high-fidelity speech as WaveNet, while synthesizing several orders of magnitude faster as it only requires a few sequential steps to generate very long waveforms with hundreds of thousands of time-steps. Furthermore, it can significantly reduce the likelihood gap that has existed between autoregressive models and flow-based models for efficient synthesis. Finally, our small-footprint WaveFlow has only 5.91M parameters, which is 15$\times$ smaller than WaveGlow. It can generate 22.05 kHz high-fidelity audio 42.6$\times$ faster than real-time (at a rate of 939.3 kHz) on a V100 GPU without engineered inference kernels.
1. Introduction
Generating high-fidelity raw audio is a computationally demanding task that has historically forced a choice between quality and efficiency. Autoregressive (AR) models, such as WaveNet, produce high-quality audio but are slow due to their sequential nature. Flow-based models, like WaveGlow, allow for parallel synthesis but often require a massive number of parameters to match AR quality. Additionally, a "likelihood gap" persists where flow-based models struggle to fit the data distribution as effectively as AR models.
WaveFlow addresses these challenges by introducing a compact architecture that bridges the gap between these two paradigms. By reshaping 1-D audio into a 2-D matrix and applying local autoregression, WaveFlow achieves state-of-the-art quality with a significantly reduced parameter count and high inference speed.
2. Methods
WaveFlow's architecture is built around the concept of balancing parallelism with the expressivity of autoregressive modeling.
2.1 The 2-D Squeeze and Local Autoregression
The model employs a "2-D squeeze" operation, reshaping the 1-D waveform into a 2-D matrix with $h$ rows and $w$ columns. Unlike standard squeezing operations that can destroy temporal structure, this method preserves the sequence while enabling a novel dependency structure. WaveFlow is designed to be autoregressive specifically over the height ($h$) dimension.
As shown in Figure 2, the model uses a stack of 2-D dilated convolutions to capture long-range dependencies across the width (time) dimension while modeling local variations across the height dimension. This approach allows the model to generate waveforms in only $h$ sequential steps, where $h$ is typically small (e.g., 8 or 16), providing a massive speedup over fully sequential models.
2.2 A Unified Framework for 1-D Generative Models
WaveFlow provides a unified framework that generalizes existing likelihood-based models. By adjusting the height $h$, the model can transition between different architectural extremes.
When $h$ equals the full length of the audio, the model behaves as a standard AR model (Figure 1a). When $h=1$, it functions as a bipartite flow similar to WaveGlow (Figure 1b). WaveFlow leverages this flexibility to achieve the high log-likelihood of AR models while maintaining the synthesis efficiency of flow-based models.
3. Experimental Results
The model was evaluated on the LJ Speech dataset (22.05 kHz) and a proprietary 24 kHz female speech dataset.
3.1 Efficiency and Compactness
WaveFlow demonstrates remarkable efficiency. The small-footprint version uses only 5.91M parameters, making it 15$\times$ smaller than WaveGlow (87.88M parameters). Despite its small size, it synthesizes 22.05 kHz audio 42.6$\times$ faster than real-time on an NVIDIA V100 GPU without specialized inference kernels.
3.2 Audio Quality and Likelihood
In Mean Opinion Score (MOS) evaluations, WaveFlow achieved a score of 4.32, comparable to WaveGlow (4.34) and approaching WaveNet (4.43). Furthermore, WaveFlow effectively closed the likelihood gap, achieving a test log-likelihood of 5.055, nearly identical to WaveNet's 5.059. This indicates that the architecture is highly effective at capturing the true distribution of human speech.
4. Conclusion
WaveFlow offers a compact and efficient solution for raw audio generation. By treating audio as a 2-D signal and utilizing local autoregression, it combines the high fidelity of autoregressive models with the synthesis speed of flow-based models. Its small footprint and high performance make it a strong candidate for real-world speech synthesis applications and a potential framework for other sequential data domains.
Summary updated