TL;DR
WaveFlow is a compact, flow-based generative model for raw audio that bridges the gap between high-fidelity but slow autoregressive models (like WaveNet) and fast but massive flow models (like WaveGlow). By using a novel 2-D dilated convolutional architecture that is autoregressive over a "squeezed" height dimension, it achieves state-of-the-art speech quality with a 15x smaller footprint than WaveGlow and synthesis speeds over 40x faster than real-time.
The Challenge of Raw Audio Generation
Generating high-fidelity raw audio is a major challenge in deep learning. Existing solutions generally fall into three camps, each with a significant drawback:
- Autoregressive (AR) models (e.g., WaveNet): These provide the highest audio quality and likelihood scores but are prohibitively slow because they generate audio sequentially, sample-by-sample.
- Bipartite Flow models (e.g., WaveGlow): These allow for fast parallel synthesis but require an enormous number of parameters (often 80M+) to match the quality of AR models. They also tend to lose temporal information by "squeezing" audio into channels too aggressively.
- Distillation-based models (e.g., Parallel WaveNet): These are fast but require a complex training pipeline involving a pre-trained "teacher" model and multiple auxiliary loss functions.
The authors of WaveFlow sought a model that is easy to train (using simple maximum likelihood), fast to sample, high-fidelity, and has a small memory footprint suitable for on-device deployment.
The WaveFlow Architecture
WaveFlow introduces a unified framework that treats WaveNet and WaveGlow as special cases. Its core innovation lies in how it structures the flow transformation:
- 2-D Squeezing: Instead of squeezing 1-D audio into many channels, WaveFlow reshapes the 1-D waveform into an $h \times w$ 2-D matrix.
- Local Autoregression: The model is designed to be autoregressive only over the height dimension ($h$). This means to generate a full matrix, it only needs $h$ sequential steps. Since $h$ is typically small (e.g., 8 or 16), this is orders of magnitude faster than the thousands of steps required by standard AR models.
- 2-D Dilated Convolutions: It uses a stack of 2-D dilated convolutions to handle long-range dependencies in the audio while modeling local variations expressively.
- Unified View: By adjusting the height $h$, WaveFlow can trade off between model capacity and synthesis parallelism. If $h=1$, it behaves like a bipartite flow; if $h=n$ (the full length), it becomes a standard AR model.
Key Insights and Receptive Fields
- The "Likelihood Gap": The paper demonstrates that the performance gap between AR and flow models is largely due to the limited expressiveness of bipartite transformations. By introducing a small amount of sequentiality (the height $h$), WaveFlow significantly closes this gap.
- Receptive Field Matters: The authors found that for good results, the receptive field over the height dimension must be at least as large as the height $h$ itself. This ensures the model doesn't introduce unnecessary conditional independence.
- Bidirectional Modeling via Permutation: Stacking multiple flows with different permutation strategies (like reversing the height order) allows the model to look at both "past" and "future" samples, which is crucial for high-fidelity synthesis.
Key Results
The researchers evaluated WaveFlow on two main datasets: LJ Speech (for primary likelihood and quality benchmarks) and a proprietary TTS dataset (to test WaveFlow as a vocoder within a full text-to-speech pipeline).
- Compactness: A WaveFlow model with only 5.91M parameters achieved comparable audio quality to a WaveGlow model with 268M parameters.
- Speed: It synthesizes 22.05 kHz audio 42.6x faster than real-time on a single V100 GPU, without needing specialized, highly-engineered inference kernels.
- Quality: In Mean Opinion Score (MOS) tests, WaveFlow scored 4.32, very close to the ground truth (4.56) and standard WaveNet (4.43), and significantly higher than smaller versions of WaveGlow.
- Likelihood: WaveFlow achieved higher test log-likelihoods than WaveGlow, confirming it captures the data distribution more accurately.
Limitations and Future Work
While WaveFlow is much faster than WaveNet, it is not fully parallel like WaveGlow; it still requires $h$ sequential steps. Although $h$ is small, this could be a bottleneck for extremely low-latency requirements.
The authors suggest that future work could involve exploring different 2-D architectures or more sophisticated permutation strategies to further reduce the number of sequential steps while maintaining the high likelihood scores that make WaveFlow so efficient.
Summary updated