Bayesian Model Comparison and Backprop Nets

David J. C. MacKayNeurIPS 1991

This 1992 paper by David J.C. MacKay is a foundational text in Bayesian machine learning. It provides a rigorous mathematical framework for applying Occam’s Razor to neural networks, allowing researchers to objectively choose model architectures and regularization parameters without relying solely on trial-and-error or massive cross-validation.

TL;DR

MacKay demonstrates that Bayesian inference naturally embodies "Occam’s Razor," penalizing overly complex models without the need for ad-hoc penalty terms. By calculating the "evidence" for a model, we can objectively determine the optimal number of hidden units and weight decay rates using only the training data, while also gaining quantified error bars on the network's predictions.

What Problem Does This Paper Address?

In the early 90s, training "backprop" (feedforward) neural networks was often a "black art." Engineers struggled with two main questions:

  1. Model Selection: How many hidden units should a network have? Too few, and it can’t learn the data; too many, and it overfits.
  2. Regularization: How much "weight decay" (penalty on large weights) is needed to prevent overfitting?

The standard approach was cross-validation—setting aside a portion of data to test different configurations. However, this is computationally expensive and "wastes" data that could otherwise be used for training. MacKay sought a principled, probabilistic way to make these choices using the training data itself.

Methods

The paper introduces a two-level inference framework:

  • Level 1: Model Fitting. Given a specific model (e.g., a network with 10 hidden units), we find the most probable weights ($w_{MP}$) by minimizing an objective function. This is what most people think of as "training."
  • Level 2: Model Comparison. We rank different models by calculating the "Evidence"—the probability of the data given the model $P(D|\mathcal{H}_i)$.

The Occam Factor

The core innovation is the decomposition of the Evidence into two parts:

  1. Best-fit Likelihood: How well the model can actually fit the data.
  2. The Occam Factor: A penalty term that measures how much of the model's parameter space was "shrunk" by the data.
Illustration of the Occam factor: the posterior distribution shrinks compared to the prior.

A complex model has a vast "prior" space of possible weight configurations. If only a tiny, finely-tuned region of that space fits the data, the Occam Factor becomes very small, penalizing the model. Simple models have a larger Occam Factor because they are less "flexible" and thus more likely to be correct if they fit the data at all.

The Hessian and $\gamma$

MacKay uses a Gaussian approximation (Laplace's method) around the best-fit weights. This involves calculating the Hessian matrix (the second derivatives of the error function). From this, he derives $\gamma$, the "effective number of parameters" determined by the data, which is often much smaller than the actual number of weights in the network.

Datasets

The paper uses a noisy interpolation problem (synthetic data) to demonstrate the concepts.

  • A hundred different networks of varying complexity were trained on the same data.
  • The results compare "Training Error" and "Test Error" against the "Log Evidence" calculated by the Bayesian framework.

Key Insights and Ideas

  • Occam’s Razor is Automatic: You don't need to manually add "complexity penalties." If you follow Bayes' Rule correctly, the math naturally favors the simplest model that explains the data.
  • Evidence Predicts Generalization: The most striking insight is that the "Evidence" (calculated only on training data) correlates remarkably well with the "Test Error" (performance on unseen data).
  • Quantified Uncertainty: Unlike standard backprop, which gives a single point estimate, this framework provides "error bars" for both the weights and the final output of the network.
  • Unified Theory: MacKay links his Bayesian approach to other statistical theories like Moody’s Generalized Prediction Error (GPE) and the V-C dimension, showing that $\gamma$ (effective parameters) is the common thread between them.

Key Results

  • Validation of the Evidence Framework: The paper demonstrates a near-perfect correlation between the Log Evidence and the Test Error. This proves that the Bayesian framework can successfully identify the model that will generalize best without needing a hold-out validation set.
Correlation between Log Evidence and Test Error.
  • Objective Regularization: The paper provides formulas to "re-estimate" the optimal weight decay ($\alpha$) and noise levels ($\beta$) on the fly during training, removing the need for manual tuning.
  • Computational Efficiency: MacKay argues that while calculating the Hessian matrix is an extra step, it is no more demanding than the task of finding the best-fit parameters and their error bars in the first place.

Limitations and Future Work

  • The Gaussian Assumption: The framework relies on the posterior distribution being "well-behaved" and peaky (Gaussian). In very non-linear or "multimodal" landscapes (common in deep learning), this approximation might break down.
  • Computational Scaling: While manageable for the networks of 1992, calculating the full Hessian matrix for modern "Deep Learning" models with millions of parameters is computationally prohibitive (though this led to future work on diagonal or Kronecker-factored approximations).
  • Integration with Decision Theory: The paper notes that "inference" (finding probabilities) is distinct from "decision" (taking action). Future work would need to bridge how these probabilistic rankings should lead to specific engineering choices in a production pipeline.

Summary updated

BibTeX
@inproceedings{mackay1991bayesian,
  author = {David J. C. MacKay},
  title = {{Bayesian Model Comparison and Backprop Nets}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
}
Figure