Autoregressive Generative Modeling with Noise Conditional Maximum Likelihood Estimation

Henry Li, Yuval Kluger

TL;DR

Autoregressive image models trained via Maximum Likelihood Estimation (MLE) are notoriously brittle: imperceptible 1-bit noise causes their likelihoods to collapse, and compounding sampling errors degrade visual quality. This paper introduces Noise Conditional Maximum Likelihood (NCML), an objective that trains autoregressive models across a continuum of noise scales. This preserves exact likelihood evaluation while enabling score-based reverse-diffusion refinement during sampling, slashing CIFAR-10 generation FID from 37.50 to 12.09 and achieving state-of-the-art likelihoods on ImageNet 64×64.

What Problem Does This Paper Address?

While MLE provides strong theoretical properties (asymptotic consistency, efficiency, and exact $O(1)$ density evaluation), standard likelihood models suffer from two major flaws:

  1. The Likelihood–Quality Disconnect: Maximizing log-likelihood often prioritizes modeling imperceptible, high-frequency noise rather than salient semantic content. Flipping just the least significant bit (LSB) in an 8-bit image—completely invisible to humans—causes standard models' negative log-likelihoods to skyrocket.
  2. Autoregressive Exposure Bias (Covariate Shift): Autoregressive models generate images token by token without an opportunity to revisit earlier pixels. Any small sampling error shifts the distribution, compounding errors downstream and leading to incoherent, degraded samples.

Diffusion models address sequential error correction via iterative denoising, but they sacrifice exact closed-form likelihoods, require hundreds or thousands of network evaluations, and lose standard MLE asymptotic guarantees. The authors look for a unified middle ground.

Methods

The authors introduce Noise Conditional Maximum Likelihood (NCML) and a paired two-stage sampling routine:

  • NCML Objective: Rather than maximizing a single log-likelihood $\mathbb{E}_{x}[\log p_\theta(x)]$, NCML optimizes an expectation over continuous noise scales $t \sim \mu$: $$\mathbb{E}_{t \sim \mu} \, \mathbb{E}_{x \sim p_t}[\log p_{\theta,t}(x)]$$ This acts as a principled, data-dependent regularizer that preserves standard MLE guarantees while penalizing hypersensitivity to local perturbations.

  • Architecture (NCPN): The Noise Conditional Pixel Network augments a PixelCNN++ backbone with axial self-attention layers after each residual block and injects noise-scale embeddings via Gaussian Fourier projection layers.

  • Two-Stage Generation (Noisy Sampling + Score-Based Refinement):
  • Noisy Autoregressive Sampling: Sample an initial image coordinate-by-coordinate from a moderately noisy scale $p_{\theta, t}$ ($t > 0$), where the distribution is smooth and autoregressive drift is suppressed.
  • Reverse-Diffusion Refinement: Because the network models exact log-probabilities, the Stein score $\nabla_x \log p_t(x)$ can be computed directly via backpropagation. The noisy draft is then polished down to $t=0$ by simulating a reverse-time SDE (using Variance Preserving or Variance Exploding schedules) without needing to start from pure Gaussian noise.

Datasets

The authors evaluated their approach on standard image generation benchmarks:

  • CIFAR-10 (unconditional, class-conditional, image completion, and out-of-distribution detection)
  • ImageNet 64×64 (unconditional density estimation)
  • CelebA 64×64 (unconditional face generation)
  • OOD Benchmarks: SVHN, Constant Uniform noise, and Random Uniform noise.
  • Corrupted Sanity Benchmarks: Perturbed variants $p_\pi$ of CIFAR-10 and ImageNet where the least significant bit (LSB) of each pixel is flipped with probability $\pi \in \{0.5, 1.0\}$.

Key Insights and Ideas

  • The "Least Significant Bit" Trap: At $\pi = 0.5$ (where the least significant bit is completely randomized), prior likelihood models collapse to roughly the same performance (~3.75–3.85 BPD). This indicates that historical benchmark improvements in standard likelihood modeling were predominantly capturing imperceptible LSB information rather than semantic structure.
  • Free Score Matching via Exact Likelihoods: While diffusion models estimate scores without modeling the underlying density directly, an explicit likelihood model naturally yields the score function $\nabla_x \log p_{\theta,t}(x)$ by taking the gradient with respect to the input.
  • Implicit Regularization at $t=0$: Even when conditioned on the unperturbed data ($t=0$), NCML-trained models demonstrate superior robustness to noise corruptions compared to vanilla models, indicating that multi-scale training acts as an effective inductive bias on finite-capacity neural networks.

Key Results

  • Generation Quality: On CIFAR-10, NCPN with a Variance Preserving schedule achieved an FID of 12.09, dramatically outperforming prior autoregressive likelihood models (e.g., Sparse Transformer at 37.50 and PixelSNAIL at 36.62).
  • Density Estimation: On ImageNet 64×64, NCPN (NCML-VE) achieved state-of-the-art likelihood among explicit likelihood models at 3.32 bits per dimension (BPD) (surpassing DenseFlow at 3.35 BPD and Sparse Transformer at 3.44 BPD).
  • Robustness Under Noise: Under full LSB corruption ($\pi=1.0$), NCPN models retained superior likelihoods (e.g., 3.89 BPD on CIFAR-10 compared to $\ge 3.97$ BPD for standard baselines).
  • Out-of-Distribution (OOD) Detection: On CIFAR-10 vs. SVHN/Uniform, NCPN attained an average AUROC of 0.80, markedly improving upon vanilla PixelCNN++ (0.44) and energy-based models (0.64).

Limitations and Future Work

  • Residual Gap to Human Perception: While substantially more resilient than standard MLE models, NCML models still show non-negligible likelihood variance under imperceptible perturbations, showing that models remain somewhat disconnected from human perceptual invariance.
  • Heuristic Noise Horizons: The maximum diffusion times ($T$) were chosen via simple heuristics (matching a standard deviation of $\approx 10$ bits per pixel). Systematically tuning noise distributions and schedules could yield better generation-likelihood trade-offs.
  • Autoregressive Overhead: Although the score-based refinement phase requires far fewer steps than initializing diffusion from pure noise, generating the initial draft still relies on sequential token-by-token autoregressive sampling.

Summary updated

Abstract

We introduce a simple modification to the standard maximum likelihood estimation (MLE) framework. Rather than maximizing a single unconditional likelihood of the data under the model, we maximize a family of \textit{noise conditional} likelihoods consisting of the data perturbed by a continuum of noise levels. We find that models trained this way are more robust to noise, obtain higher test likelihoods, and generate higher quality images. They can also be sampled from via a novel score-based sampling scheme which combats the classical \textit{covariate shift} problem that occurs during sample generation in autoregressive models. Applying this augmentation to autoregressive image models, we obtain 3.32 bits per dimension on the ImageNet 64x64 dataset, and substantially improve the quality of generated samples in terms of the Frechet Inception distance (FID) --- from 37.50 to 12.09 on the CIFAR-10 dataset.
BibTeX
@inproceedings{li2023autoregressive,
  author = {Henry Li and Yuval Kluger},
  title = {{Autoregressive Generative Modeling with Noise Conditional Maximum Likelihood Estimation}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=r0JMLPgGXS_},
}

Loading related content...

Figure