Introduction
In Bayesian Deep Learning, practitioners typically place a prior distribution—usually Gaussian—over the weights of a neural network. A fundamental question is: what kind of prior does this induce on the actual outputs of the network in function space?
For decades, theorists have relied on the infinite-width limit. If a hidden layer is infinitely wide, the Central Limit Theorem dictates that the output prior becomes a simple Gaussian. However, real-world networks are finite. While finite networks excel at representation learning and adapting to data structure, their mathematics are notoriously difficult. Until now, researchers have relied on perturbative methods (like Edgeworth expansions) to approximate finite-width behavior. Unfortunately, these approximations often miss the most interesting non-Gaussian features of the network.
Recently, researchers have moved beyond these approximations to derive the first exact mathematical formulas for the output priors of finite-width Bayesian Neural Networks (BNNs). They discovered that while infinite networks are Gaussian, finite networks possess "heavy-tailed" distributions that become increasingly extreme with depth—a feature standard approximation methods completely fail to capture.
Methods: BNNs as Exactly Solvable Models
Treating BNNs as "exactly solvable models"—a concept borrowed from theoretical physics—the authors focused on fully-connected feedforward networks (both linear and ReLU) without bias terms.
Their technical approach involved several key innovations:
- Characteristic Functions: Instead of working directly with probability densities, they solved for the Fourier transform of the prior.
- Meijer G-functions: For deep linear networks, the prior can be expressed exactly using the Meijer G-function, a "master" special function that encompasses almost all other common functions (like exponentials and Bessel functions).
- ReLU as a Mixture: They proved that the prior of a ReLU network is actually a weighted mixture of the priors of linear networks with smaller widths. This corresponds to the different combinations of "active" versus "inactive" neurons in each layer.
Because this is a theoretical paper focused on exact mathematical identities, the authors validated their formulas using numerical simulations rather than standard training datasets. They sampled weights for millions of random networks ($10^8$ examples) and compared the empirical distribution of these outputs to their derived formulas, finding a perfect match.
Key Insights and Results
The derivations yielded several major insights into the behavior of finite BNNs:
- The "Heavy Tail" Reality: As networks get deeper or narrower, their output priors develop "heavy tails" (sub-Weibull distributions). This means the network is much more likely to produce extreme, outlier outputs than a Gaussian model would predict.
- Depth vs. Width: Increasing depth makes the distribution more non-Gaussian (heavier tails), while increasing width pulls it back toward a Gaussian shape.
- The Failure of Perturbation Theory: Standard "Edgeworth expansions" (the traditional tool for finite-width analysis) are shown to be "thin-tailed." While they might accurately model the middle of the distribution, they fundamentally cannot capture the heavy tails that exist in finite networks.
- Exact Formulas and Moments: The paper provides the first closed-form expressions for the marginal prior of deep linear and ReLU networks. For a simple two-layer linear network, the prior is exactly a modified Bessel function of the second kind. They also derived exact formulas for all moments of the output norm, showing how variance and higher-order statistics scale. For example, a ReLU network's variance is exactly $2^{1-d}$ times that of a linear network of the same architecture.
Limitations and Future Work
While a significant theoretical leap, the work has constraints:
- Single-Input Only: The current formulas describe the output distribution for a single input point. In practice, the joint distribution of multiple inputs (covariance between data points) is crucial.
- No Bias Terms: The derivations assume networks have no bias parameters, as adding biases breaks the rotational symmetry that makes the current math tractable.
- Architecture Constraints: The exact solutions are limited to fully-connected feedforward architectures. Extending this framework to Convolutional Neural Networks (CNNs) or Transformers remains a significant open challenge.
Ultimately, these exact solutions provide a new "gold standard" for checking the validity of future approximate theories in Bayesian deep learning.
Summary updated