SampleRNN: An Unconditional End-to-End Neural Audio Generation Model

Soroush Mehri, Kundan Kumar, Ishaan Gulrajani, Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron C. Courville, Yoshua Bengio

Abstract

In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.

Introduction

Generating high-fidelity audio presents a significant "curse of dimensionality." Standard audio is typically sampled at 16kHz or higher, meaning a single second of audio contains at least 16,000 data points. Traditional Recurrent Neural Networks (RNNs) struggle at this resolution; modeling audio sample-by-sample requires a memory span of thousands of steps to capture even a single musical note or phoneme, leading to vanishing gradients and high computational costs.

Previous approaches often relied on lossy compression or hand-engineered features like vocoders, which frequently produced "robotic" artifacts. SampleRNN addresses this by modeling the raw waveform directly through a hierarchical neural network that nests multiple RNNs operating at different "clock rates."

Methods: Hierarchical Architecture

The core innovation of SampleRNN is its multi-tier structure, which allows the model to capture both fine-grained textures and long-term structural patterns simultaneously.

The 3-tier SampleRNN architecture showing hierarchical frame-level and sample-level processing.
  • Frame-Level Tiers (Top/Middle): These tiers operate at lower resolutions, processing blocks of samples (frames) rather than individual points. Higher tiers have slower "clock rates," using deep RNNs (specifically GRUs or LSTMs) to summarize audio history into a conditioning vector.
  • Sample-Level Tier (Bottom): This tier operates at the actual sample rate. Interestingly, the authors found that a memory-less Multilayer Perceptron (MLP) performs best here, taking the conditioning vector from higher tiers and the immediate preceding samples to predict the next single sample.
  • Upsampling and Quantization: Linear projections bridge the gap between tiers. To handle the complexity of audio distributions, the model discretizes audio into 256 levels (8-bit) and treats generation as a classification problem using a Softmax output. This discrete approach outperformed continuous models like Gaussian Mixture Models (GMMs).

Experiments and Results

The model was evaluated on three distinct datasets: Blizzard (speech synthesis), Onomatopoeia (human vocal sounds), and Music (Beethoven piano sonatas).

Comparison of real audio waveforms with samples generated by 2-tier and 3-tier SampleRNN models across three datasets.

Quantitative and Qualitative Performance

SampleRNN achieved lower Negative Log-Likelihood (NLL) scores than standard RNNs and a re-implementation of WaveNet across all datasets. For instance, on the Blizzard dataset, the 3-tier SampleRNN achieved an NLL of 1.387 compared to 1.434 for a standard RNN. In "AB" preference tests, human listeners overwhelmingly preferred samples from the 3-tier SampleRNN over baselines.

Key Insights

  • Sample-Level Autoregression: Experiments proved that each sample must be conditioned on the one immediately preceding it; predicting blocks of samples simultaneously resulted in a total loss of coherence.
  • Memory Span: In "silence injection" tests, the model demonstrated an ability to "remember" speaker characteristics. After a full second of silence, the model resumed generating audio matching the original speaker's identity in 83% of cases.

Conclusion

SampleRNN successfully models raw audio by leveraging a hierarchy of temporal scales. While the autoregressive nature of the model makes inference computationally expensive, its ability to capture long-term dependencies without hand-engineered features marks a significant step forward in unconditional audio generation. Future work involves adapting this architecture for conditional tasks such as text-to-speech or genre-specific music generation.

Summary updated

Abstract

In this paper we propose a novel model for unconditional audio generation based on generating one audio sample at a time. We show that our model, which profits from combining memory-less modules, namely autoregressive multilayer perceptrons, and stateful recurrent neural networks in a hierarchical structure is able to capture underlying sources of variations in the temporal sequences over very long time spans, on three datasets of different nature. Human evaluation on the generated samples indicate that our model is preferred over competing models. We also show how each component of the model contributes to the exhibited performance.
BibTeX
@inproceedings{mehri2017samplernn,
  author = {Soroush Mehri and Kundan Kumar and Ishaan Gulrajani and Rithesh Kumar and Shubham Jain and Jose Sotelo and Aaron C. Courville and Yoshua Bengio},
  title = {{SampleRNN: An Unconditional End-to-End Neural Audio Generation Model}},
  booktitle = {International Conference on Learning Representations},
  year = {2017},
  url = {https://openreview.net/forum?id=SkxKPDv5xl},
}
Figure