Abstract
Autoregressive models use the chain rule to define a joint probability distribution as a product of conditionals. These conditionals need to be normalized, imposing constraints on the functional families that can be used. To increase flexibility, we propose autoregressive conditional score models (AR-CSM) where we parameterize the joint distribution in terms of the derivatives of univariate log-conditionals (scores), which need not be normalized. To train AR-CSM, we introduce a new divergence between distributions named Composite Score Matching (CSM). For AR-CSM models, this divergence between data and model distributions can be computed and optimized efficiently, requiring no expensive sampling or adversarial training. Compared to previous score matching algorithms, our method is more scalable to high dimensional data and more stable to optimize. We show with extensive experimental results that it can be applied to density estimation on synthetic data, image generation, image denoising, and training latent variable models with implicit encoders.
Introduction
Generative modeling often involves a trade-off between flexibility and tractability. Autoregressive (AR) models are easy to evaluate but restricted by the requirement that each conditional component must be a normalized probability distribution. Conversely, Energy-Based Models (EBMs) offer immense flexibility but are notoriously difficult to train due to the intractable partition function.
Score Matching (SM) was developed to train EBMs by matching the gradient of the log-density (the "score") rather than the density itself, bypassing the partition function. However, traditional SM requires calculating the trace of the Hessian matrix, which is computationally prohibitive in high dimensions. In this work, we introduce Composite Score Matching (CSM) and Autoregressive Conditional Score Models (AR-CSM) to overcome these limitations.
Methods
Composite Score Matching (CSM)
Instead of matching the joint distribution's score directly, CSM leverages the chain rule to decompose the problem into matching univariate conditional scores: $$s_{\theta,d}(x_{<d}, x_d) = \frac{\partial}{\partial x_d} \log q_\theta(x_d | x_{<d})$$ By applying integration by parts to these univariate terms, we derive a tractable objective that requires only first-order derivatives. This eliminates the $O(D)$ Hessian trace computation, making the method significantly more scalable.
Autoregressive Conditional Score Models (AR-CSM)
AR-CSM utilizes standard autoregressive backbones (such as PixelCNN++ or MADE) to extract context from previous dimensions ($x_{<d}$). This context informs a neural network that outputs the score for the current dimension ($x_d$). Crucially, because we model scores rather than densities, the output does not need to be normalized, allowing for more expressive architectures.
Inference via Langevin Dynamics
Data generation is performed using one-dimensional Langevin dynamics. The model samples dimensions sequentially ($x_1$, then $x_2$, etc.). Because the Langevin chain operates in a univariate space for each step, it converges much faster than high-dimensional joint sampling.
Key Results
Scalability and Stability
Experimental results demonstrate that CSM is significantly more efficient than traditional Score Matching. While SM training time grows linearly with data dimension and faces memory exhaustion at $D \approx 200$, CSM remains stable and computationally efficient across high dimensions. Furthermore, as a deterministic objective, CSM exhibits near-zero variance compared to stochastic methods like Sliced Score Matching (SSM).
Training Dynamics
The stability of the CSM objective is further evidenced by training loss curves. Compared to Denoising Score Matching (DSM) and Sliced Score Matching (SSM), CSM converges more rapidly and maintains a much smoother loss profile, indicating a more stable optimization landscape.
Performance Benchmarks
- Out-of-Distribution (OOD) Detection: AR-CSM achieved an AUROC of 0.73 on CIFAR-10 vs. SVHN tasks, significantly outperforming PixelCNN++ (0.44) and Glow (0.41).
- Variational Autoencoders: CSM-trained VAEs with implicit encoders achieved superior negative log-likelihoods on MNIST (95.02) compared to standard ELBO methods (96.74).
Conclusion
Composite Score Matching provides a scalable, stable, and flexible framework for high-dimensional density estimation. By focusing on univariate conditional scores, AR-CSM bypasses the normalization constraints of traditional autoregressive models and the computational bottlenecks of standard score matching. Future work will address the sequential nature of sampling to improve generation speeds for high-resolution data.
Summary updated