Autoregressive Score Matching

Chenlin Meng, Lantao Yu, Yang Song, Jiaming Song, Stefano Ermon

Generative modeling often forces researchers to choose between flexibility and tractability. This paper introduces Autoregressive Conditional Score Models (AR-CSM), a novel approach that trains flexible, high-dimensional generative models without requiring normalized probability distributions. By combining the sequential structure of autoregressive models with the computational efficiency of score matching, AR-CSM achieves greater scalability than traditional score matching and higher expressivity than standard autoregressive models.

The Trade-off in Generative Modeling

Historically, generative models have struggled to balance two competing needs:

  1. Tractability (Autoregressive Models): Models like PixelCNN are highly tractable because they break a joint distribution into a product of conditionals using the chain rule. However, these conditionals must be normalized (sum to 1). This strict constraint often forces the use of simple functional families (like Gaussians), which may fail to capture complex, multi-modal data distributions.
  2. Flexibility (Energy-Based Models & Score Matching): Energy-Based Models (EBMs) are highly flexible because they do not require normalization. However, calculating their normalization constant (the partition function) is often intractable. Score Matching (SM) bypasses this by matching the gradient of the log-density (the "score"). Unfortunately, standard SM requires calculating the trace of a Hessian matrix, an operation whose computational cost scales prohibitively as data dimensionality increases.

Composite Score Matching (CSM)

The authors bridge this gap by proposing a new training objective: Composite Score Matching (CSM).

Instead of attempting to match the score of the entire joint distribution simultaneously, CSM leverages the chain rule of probability to decompose the problem into a sequence of univariate (one-variable) conditional scores.

This yields several key advantages:

  • Bypassing the Hessian: The CSM divergence measure only requires the derivatives of univariate log-conditionals. This completely eliminates the expensive Hessian trace calculation that bottlenecks standard Score Matching.
  • AR-CSM Architecture: The authors adapt existing autoregressive architectures (like MADE or PixelCNN++). Instead of outputting parameters for a normalized distribution, these networks output a "context vector" fed into a smaller network to produce the unnormalized conditional score.
  • Efficient Sampling: To generate samples, the model employs 1D Langevin dynamics (a gradient-based sampling method) sequentially across each dimension, following the autoregressive order.

Key Insights and Results

The researchers evaluated AR-CSM across several benchmarks, including synthetic 2D data, MNIST, CIFAR-10, CelebA, and SVHN. The results highlight significant improvements in scalability, stability, and expressivity.

Scalability and Stability

The most striking advantage of CSM is its computational efficiency. In tests on multivariate Gaussians, the training time for standard Score Matching (SM) grew linearly with data dimensionality, eventually crashing due to memory limits. In contrast, CSM's training time remained nearly constant.

Furthermore, unlike Sliced Score Matching (SSM)—which relies on random projections to estimate scores and suffers from high variance—CSM is a deterministic objective. This results in zero variance during optimization and vastly superior training stability.

Training time and variance comparison between CSM, SM, and SSM.

This stability is also evident in the training loss curves. When compared to Denoising Score Matching (DSM) and SSM, CSM converges much faster and with significantly less noise.

Training loss convergence comparison across DSM, SSM, and CSM.

Expressivity and Sample Quality

Because AR-CSM's conditional scores do not have to correspond to a normalized distribution, the model can capture much more complex relationships between variables. AR-CSM models consistently produced sharper, higher-quality images compared to their maximum-likelihood (MLE) counterparts. For instance, an AR-CSM based on the MADE architecture generated significantly better MNIST digits than a standard MADE model.

Additionally, when applied to train Variational Autoencoders (VAEs) with "implicit" encoders, CSM achieved better Likelihood and FID scores than alternative score-estimation methods like Stein or Spectral matching.

Out-of-Distribution Detection

An interesting byproduct of the AR-CSM architecture is its utility in anomaly detection. The authors found that the sum of the conditional scores serves as a highly effective statistic for detecting Out-of-Distribution (OOD) data, outperforming several established baselines.

Limitations and Future Directions

The primary limitation of AR-CSM is its sampling speed. Because the model relies on an autoregressive structure, it must generate data sequentially (e.g., pixel by pixel). While 1D Langevin dynamics is faster than high-dimensional Langevin approaches, the sequential requirement still makes sampling slower than "one-shot" generative models like GANs or certain Flow-based models.

Future work may focus on accelerating this sampling procedure or exploring dynamic, data-driven variable orderings to further optimize the generation process.

Summary updated

Abstract

Autoregressive models use chain rule to define a joint probability distribution as a product of conditionals. These conditionals need to be normalized, imposing constraints on the functional families that can be used. To increase flexibility, we propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariate log-conditionals (scores), which need not be normalized. To train AR-CSM, we introduce a new divergence between distributions named Composite Score Matching (CSM). For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. Compared to previous score matching algorithms, our method is more scalable to high dimensional data and more stable to optimize. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
BibTeX
@inproceedings{meng2020autoregressive,
  author = {Chenlin Meng and Lantao Yu and Yang Song and Jiaming Song and Stefano Ermon},
  title = {{Autoregressive Score Matching}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=83wpag_rIod},
}
Figure