Abstract
This paper introduces an alternative approach to sampling from autoregressive models. Autoregressive models are typically sampled sequentially, according to the transition dynamics defined by the model. Instead, we propose a sampling procedure that initializes a sequence with white noise and follows a Markov chain defined by Langevin dynamics on the global log-likelihood of the sequence. This approach parallelizes the sampling process and generalizes to conditional sampling. Using an autoregressive model as a Bayesian prior, we can steer the output of a generative model using a conditional likelihood or constraints. We apply these techniques to autoregressive models in the visual and audio domains, with competitive results for audio source separation, super-resolution, and inpainting.
Introduction
Autoregressive (AR) models, such as WaveNet for audio or PixelCNN for images, are among the most powerful generative models available today. By predicting each data point (pixel or audio sample) based on all previous ones, they capture intricate dependencies. However, they face two significant challenges: the sequential bottleneck and inflexibility. Standard "ancestral sampling" is strictly linear, requiring millions of sequential steps for high-resolution audio, which makes generation painfully slow. Furthermore, these models are typically "one-way streets," making it difficult to perform inverse tasks like inpainting or super-resolution without retraining task-specific models.
We introduce Parallel and Flexible (PnF) sampling, a novel framework that generates data using Langevin dynamics. By treating the AR model as a Bayesian prior and navigating its log-likelihood surface via gradients, PnF enables massive parallelization and allows a single pre-trained model to solve diverse inverse problems.
Methodology: Langevin Dynamics on AR Models
The core innovation of PnF is reframing sampling as a gradient-based optimization problem. We initialize a sequence with white noise and iteratively "nudge" it toward high-probability states defined by the AR model.
Autoregressive Smoothing
Langevin dynamics requires gradients, but many AR models operate on discrete data where gradients are undefined. To overcome this, we introduce Autoregressive Smoothing. We convolve the discrete distribution with a Gaussian kernel to create a differentiable "smoothed" density. The model is fine-tuned to predict un-noised values from noisy ones at various "temperatures" or noise levels ($\sigma$).
Annealed and Stochastic Sampling
Following the principles of annealed Langevin dynamics, the sampler begins with high noise levels to explore global structure and gradually "cools" the noise to refine local details. For extremely long sequences, such as audio, we utilize a stochastic PnF variant. This updates random overlapping patches in parallel across multiple GPUs, conceptually similar to the "Hogwild!" algorithm for parallel gradient descent, allowing the sampling process to scale with available hardware.
Experimental Setup
The authors evaluated PnF across two primary domains:
- Audio: Using the VCTK dataset (44 hours of multi-speaker speech) and the Supra Piano dataset (52 hours of piano recordings), sampled at 22kHz with 8-bit encoding.
- Images: Using the CIFAR-10 dataset for $32 \times 32$ image generation.
Results and Analysis
PnF produces samples with log-likelihoods comparable to traditional ancestral sampling while offering significant advantages in speed and versatility.
Computational Speedup
A striking finding is that the number of Langevin iterations required for convergence is empirically independent of the sequence length. This allows PnF to eventually outperform sequential sampling as sequences grow longer.
As shown in the figure above, while ancestral sampling time grows linearly with signal length, PnF (especially when using 8 GPUs) maintains a much flatter growth curve, enabling the generation of long audio signals in a fraction of the time.
The AR Model as a Versatile Prior
Because PnF uses gradients, it is easy to add a "steering" term to the update. By adding the gradient of a constraint or a conditional likelihood, the same pre-trained AR model can solve various tasks without retraining:
- Source Separation: Successfully splitting mixtures of speakers or instruments (achieving ~17 dB SI-SDR).
- Super-Resolution: Upscaling audio (up to 16x) and images (2x) by treating the low-resolution input as a constraint.
- Inpainting: Filling in missing blocks of data in both audio and image domains.
Discussion and Limitations
While PnF offers a powerful alternative for long sequences, it does have limitations. For short sequences, such as $32 \times 32$ images, the overhead of thousands of Langevin iterations makes it slower than ancestral sampling. Additionally, the method currently requires storing multiple fine-tuned versions of the model for different noise levels. Future work may focus on extending these gradient-based techniques to fully discrete domains, such as text, where defining gradients remains a challenge.
Summary updated