Regularized estimation of image statistics by Score Matching

Diederik P. Kingma, Yann LeCunNeurIPS 2010 · OpenReview

In the world of generative modeling, Maximum Likelihood Estimation (MLE) is the gold standard. However, for high-dimensional data like images, MLE often hits a wall: the "normalizing constant." Calculating this constant is computationally impossible for many complex models.

In this 2010 paper, Diederik P. Kingma and Yann LeCun take a fresh look at Score Matching (SM)—an elegant alternative to MLE that sidesteps the normalizing constant—and provide the missing ingredients needed to make it work for real-world image data.

TL;DR

The authors introduce a regularized version of Score Matching that allows the algorithm to handle discrete, "jagged" data like quantized images. They also provide an automated "double-backpropagation" method to calculate the complex gradients required for training, making Score Matching both stable and easy to implement for high-dimensional density estimation.

What Problem Does This Paper Address?

Training high-dimensional density models (like those used to understand the "language" of natural images) is notoriously difficult. While Product-of-Experts (PoE) models are powerful, training them via MLE is intractable because you can't easily compute the total probability volume.

Score Matching (SM) was proposed to solve this by matching the slope (the score) of the model's log-density to the slope of the data's log-density. Since the slope of a constant is zero, the pesky normalizing constant simply vanishes.

However, SM has two major "real-world" problems:

  1. The Smoothness Trap: The math behind SM assumes the underlying data distribution is smooth and differentiable. Real image data is quantized (pixel values 0-255) and finite, creating "spikes" that cause SM to become unstable.
  2. The Gradient Nightmare: Computing the loss for SM requires second-order derivatives. Differentiating that loss to update model weights requires even higher-order derivatives, which researchers previously had to derive manually for every new model.

Methods

The paper introduces two key innovations to make SM practical:

1. Curvature-Based Regularization

To handle non-smooth data, the authors propose a regularization term. Mathematically, this is equivalent to adding a tiny amount of Gaussian noise to the training data to "blur" the discrete points into a smooth distribution.

The resulting loss function includes a penalty on the square of the second derivative (curvature) of the model's energy function. This prevents the model from creating infinitely sharp "wells" around discrete data points.

2. Automated Double-Backpropagation

To solve the implementation hurdle, they developed an extended version of the double-backpropagation algorithm. This allows the computer to automatically calculate the necessary third-order derivatives.

Computational graph for the extended double-backpropagation algorithm.

Crucially, they show that by using a diagonal approximation of the Hessian, the computational cost scales linearly—$O(W)$—with the number of weights, making it efficient for large networks.

Key Insights and Results

The authors evaluated their approach on standard datasets like MNIST, the Berkeley Segmentation Dataset, and classic image benchmarks (e.g., Lena, Peppers).

Experimental results showing learned filters, the effect of regularization strength, and image reconstruction performance.
  • The "Sanding" Effect: Think of the regularization as "sanding down" the jagged edges of discrete data. Without it, Score Matching tries to fit a smooth sheet over a bed of needles; with it, the sheet rests on a smooth landscape. On MNIST, unregularized SM failed entirely, while the regularized version achieved competitive classification error rates (around 8.8% with a simple logistic regressor).
  • Intuitive Features: The filters learned by the regularized SM model on natural images are "Gabor-like" (detecting edges and textures at different scales). This confirms the model is capturing meaningful structural statistics of the visual world.
  • Optimization Sweet Spot: The authors found that the optimal regularization strength ($\lambda$) for maximum likelihood was the same as the $\lambda$ that produced the best classification and denoising results. This suggests the regularized loss is a true proxy for model quality.
  • Denoising and Super-Resolution: The regularized SM model outperformed Basis Rotation and performed slightly better than Contrastive Divergence (CD-1), successfully improving the Peak Signal-to-Noise Ratio (PSNR) of low-resolution and noisy images (e.g., reaching 29.76 PSNR on the Peppers image).

Limitations and Future Work

  • Diagonal Approximation: To keep the algorithm fast ($O(W)$), the authors used a diagonal approximation of the Hessian. While effective for the models tested, more complex "deep" architectures might require the full Hessian, which would be significantly more expensive.
  • Simple Super-resolution: The super-resolution task was a simplified version (averaging blocks). Testing on more sophisticated, non-linear degradation models remains a task for future research.
  • Comparison to NCE: The authors noted that Noise Contrastive Estimation (NCE) is a promising contemporary method but was too new to be included in their comparative experiments.

Summary updated

Abstract

Score Matching is a recently-proposed criterion for training high-dimensional density models for which maximum likelihood training is intractable. It has been applied to learning natural image statistics but has so-far been limited to simple models due to the difficulty of differentiating the loss with respect to the model parameters. We show how this differentiation can be automated with an extended version of the double-backpropagation algorithm. In addition, we introduce a regularization term for the Score Matching loss that enables its use for a broader range of problem by suppressing instabilities that occur with finite training sample sizes and quantized input values. Results are reported for image denoising and super-resolution.
BibTeX
@inproceedings{kingma2010regularized,
  author = {Diederik P. Kingma and Yann LeCun},
  title = {{Regularized estimation of image statistics by Score Matching}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2010},
  url = {https://openreview.net/forum?id=ILhODp1-Tv},
}
Figure