Bayesian Backpropagation Over I-O Functions Rather Than Weights

David H. WolpertNeurIPS 1993

In the realm of Bayesian machine learning, researchers and practitioners frequently rely on Maximum A Posteriori (MAP) estimation to find the parameters that are most likely given a set of training data. Traditionally, this optimization is applied to the weight vector ($w$) of a neural network.

However, a subtle but critical gap exists in this standard approach: we do not inherently care about the specific weights themselves; we care about the input-output function ($\phi$) those weights represent. Because the mapping from weights to functions is non-linear and highly redundant—meaning many different weight combinations can produce the exact same output behavior—the most probable weight vector does not necessarily correspond to the most probable input-output function.

This paper addresses this fundamental disconnect, proposing a mathematical shift in how we train Bayesian neural networks to ensure we are optimizing for the most likely behavior of the network, rather than just its internal parameters.

Bridging the Gap: The Jacobian Correction Term

To shift the optimization from weight space to function space, the paper introduces a "correction term" derived from the rules of probability density transformation. If we have a probability density over weights, $P(w)$, and we want to find the density over functions, $P(\phi)$, we must account for the Jacobian of the mapping from $w$ to $\phi$.

Instead of minimizing the standard cost function $L(w, t)$, the author proposes minimizing a modified cost function: $$L'(w, t) = L(w, t) + \ln |J_{\phi,w}(w)|$$ where $J$ is the Jacobian matrix of the weights-to-function mapping.

By adding this $\ln |J|$ term to conventional backpropagation, the network achieves true MAP input-output function estimation. The paper breaks this problem down analytically, handling different dimensionality cases (whether the number of weights is less than, greater than, or equal to the number of possible input-output pairs) and addressing the "global symmetries" of neural networks (e.g., swapping hidden neurons) to keep the mathematics tractable.

Key Insights and Theoretical Justifications

This theoretical shift provides a first-principles Bayesian justification for several common deep learning practices and reveals new behaviors in standard architectures.

The "Push-Out" Effect

In a simple perceptron model using a tanh activation function, the Jacobian correction term acts to "push" weights away from zero. Unless the training data provides very strong evidence to the contrary, the MAP function estimate will favor more "decisive" outputs than a standard MAP weight estimate.

Illustration of the "push-out" effect, where the modified cost function favors more decisive, extreme outputs.

As demonstrated in the perceptron example, this modified backpropagation finds different solutions than the standard version, often producing larger magnitude outputs for the same training data and exhibiting multiple extrema.

Automatic Minimum Description Length (MDL)

The correction term naturally favors functions with small description lengths. It essentially acts as an automatic version of the Minimum Description Length (MDL) principle, inherently biasing the model toward simpler, more compressible representations without requiring an explicit external penalty.

Justifying Pruning and Weight-Sharing

One of the most significant contributions of this framework is its theoretical validation of network compression techniques. The paper shows that certain "singular" weight configurations—specifically where weights are exactly zero (pruning) or identical to one another (weight-sharing)—actually have an "enhanced" probability in the function space. This provides a formal Bayesian reason why these techniques help models generalize better.

Layer-Specific Sensitivity

The Jacobian treats weights in different layers differently. This mathematical reality suggests that applying a one-size-fits-all prior for all weights in a network is theoretically suboptimal. Instead, weights leading into hidden neurons should be treated differently than those leading out of them.

Limitations and Future Directions

While the theoretical framework is robust, practical implementation faces hurdles. Calculating the determinant of the Jacobian is computationally "laborious" for large, deep networks. Although the author suggests approximations, such as using the Trace of the log, the computational overhead remains significant.

Furthermore, the correction term can lead to singularities where the probability approaches infinity. These must be handled by carefully modifying the weight priors to compensate, a process that requires further characterization. Finally, the current analysis assumes a fixed network architecture; extending this framework to perform model selection across different architectures represents the next logical frontier for this research.

Summary updated

BibTeX
@inproceedings{wolpert1993bayesian,
  author = {David H. Wolpert},
  title = {{Bayesian Backpropagation Over I-O Functions Rather Than Weights}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1993},
}
Figure