TL;DR
Autoregressive image models trained via Maximum Likelihood Estimation (MLE) are notoriously brittle: imperceptible 1-bit noise causes their likelihoods to collapse, and compounding sampling errors degrade visual quality. This paper introduces Noise Conditional Maximum Likelihood (NCML), an objective that trains autoregressive models across a continuum of noise scales. This preserves exact likelihood evaluation while enabling score-based reverse-diffusion refinement during sampling, slashing CIFAR-10 generation FID from 37.50 to 12.09 and achieving state-of-the-art likelihoods on ImageNet 64×64.
What Problem Does This Paper Address?
While MLE provides strong theoretical properties (asymptotic consistency, efficiency, and exact $O(1)$ density evaluation), standard likelihood models suffer from two major flaws:
- The Likelihood–Quality Disconnect: Maximizing log-likelihood often prioritizes modeling imperceptible, high-frequency noise rather than salient semantic content. Flipping just the least significant bit (LSB) in an 8-bit image—completely invisible to humans—causes standard models' negative log-likelihoods to skyrocket.
- Autoregressive Exposure Bias (Covariate Shift): Autoregressive models generate images token by token without an opportunity to revisit earlier pixels. Any small sampling error shifts the distribution, compounding errors downstream and leading to incoherent, degraded samples.
Diffusion models address sequential error correction via iterative denoising, but they sacrifice exact closed-form likelihoods, require hundreds or thousands of network evaluations, and lose standard MLE asymptotic guarantees. The authors look for a unified middle ground.
Methods
The authors introduce Noise Conditional Maximum Likelihood (NCML) and a paired two-stage sampling routine:
-
NCML Objective: Rather than maximizing a single log-likelihood $\mathbb{E}_{x}[\log p_\theta(x)]$, NCML optimizes an expectation over continuous noise scales $t \sim \mu$: $$\mathbb{E}_{t \sim \mu} \, \mathbb{E}_{x \sim p_t}[\log p_{\theta,t}(x)]$$ This acts as a principled, data-dependent regularizer that preserves standard MLE guarantees while penalizing hypersensitivity to local perturbations.
-
Architecture (NCPN): The Noise Conditional Pixel Network augments a PixelCNN++ backbone with axial self-attention layers after each residual block and injects noise-scale embeddings via Gaussian Fourier projection layers.
- Two-Stage Generation (Noisy Sampling + Score-Based Refinement):
- Noisy Autoregressive Sampling: Sample an initial image coordinate-by-coordinate from a moderately noisy scale $p_{\theta, t}$ ($t > 0$), where the distribution is smooth and autoregressive drift is suppressed.
- Reverse-Diffusion Refinement: Because the network models exact log-probabilities, the Stein score $\nabla_x \log p_t(x)$ can be computed directly via backpropagation. The noisy draft is then polished down to $t=0$ by simulating a reverse-time SDE (using Variance Preserving or Variance Exploding schedules) without needing to start from pure Gaussian noise.
Datasets
The authors evaluated their approach on standard image generation benchmarks:
- CIFAR-10 (unconditional, class-conditional, image completion, and out-of-distribution detection)
- ImageNet 64×64 (unconditional density estimation)
- CelebA 64×64 (unconditional face generation)
- OOD Benchmarks: SVHN, Constant Uniform noise, and Random Uniform noise.
- Corrupted Sanity Benchmarks: Perturbed variants $p_\pi$ of CIFAR-10 and ImageNet where the least significant bit (LSB) of each pixel is flipped with probability $\pi \in \{0.5, 1.0\}$.
Key Insights and Ideas
- The "Least Significant Bit" Trap: At $\pi = 0.5$ (where the least significant bit is completely randomized), prior likelihood models collapse to roughly the same performance (~3.75–3.85 BPD). This indicates that historical benchmark improvements in standard likelihood modeling were predominantly capturing imperceptible LSB information rather than semantic structure.
- Free Score Matching via Exact Likelihoods: While diffusion models estimate scores without modeling the underlying density directly, an explicit likelihood model naturally yields the score function $\nabla_x \log p_{\theta,t}(x)$ by taking the gradient with respect to the input.
- Implicit Regularization at $t=0$: Even when conditioned on the unperturbed data ($t=0$), NCML-trained models demonstrate superior robustness to noise corruptions compared to vanilla models, indicating that multi-scale training acts as an effective inductive bias on finite-capacity neural networks.
Key Results
- Generation Quality: On CIFAR-10, NCPN with a Variance Preserving schedule achieved an FID of 12.09, dramatically outperforming prior autoregressive likelihood models (e.g., Sparse Transformer at 37.50 and PixelSNAIL at 36.62).
- Density Estimation: On ImageNet 64×64, NCPN (NCML-VE) achieved state-of-the-art likelihood among explicit likelihood models at 3.32 bits per dimension (BPD) (surpassing DenseFlow at 3.35 BPD and Sparse Transformer at 3.44 BPD).
- Robustness Under Noise: Under full LSB corruption ($\pi=1.0$), NCPN models retained superior likelihoods (e.g., 3.89 BPD on CIFAR-10 compared to $\ge 3.97$ BPD for standard baselines).
- Out-of-Distribution (OOD) Detection: On CIFAR-10 vs. SVHN/Uniform, NCPN attained an average AUROC of 0.80, markedly improving upon vanilla PixelCNN++ (0.44) and energy-based models (0.64).
Limitations and Future Work
- Residual Gap to Human Perception: While substantially more resilient than standard MLE models, NCML models still show non-negligible likelihood variance under imperceptible perturbations, showing that models remain somewhat disconnected from human perceptual invariance.
- Heuristic Noise Horizons: The maximum diffusion times ($T$) were chosen via simple heuristics (matching a standard deviation of $\approx 10$ bits per pixel). Systematically tuning noise distributions and schedules could yield better generation-likelihood trade-offs.
- Autoregressive Overhead: Although the score-based refinement phase requires far fewer steps than initializing diffusion from pure noise, generating the initial draft still relies on sequential token-by-token autoregressive sampling.
Summary updated