When building predictive models, we face a fundamental tension: a model with more parameters can fit our training data more precisely, but it risks overfitting to random noise. Traditional statistical approaches often rely on ad-hoc regularization techniques or manual cross-validation to find the right balance.
Bayesian inference offers an elegant alternative where the mathematics of probability theory automatically penalize unnecessary complexity—a phenomenon known as the Bayesian Occam's Razor.
The Intuition: Conservation of Probability
To understand why this penalty happens automatically, we have to look at the marginal likelihood (also called the evidence). The marginal likelihood is the probability of observing a dataset $D$ given a specific model structure $M$, integrating over all possible parameter settings $\theta$:
$$p(D|M) = \int p(D|\theta, M) p(\theta|M) d\theta$$
The core intuition relies on a simple fact: probability mass must sum to one. Because $p(D|M)$ is a probability distribution over all possible datasets that could be observed, a model must allocate its limited probability mass across the outcomes it can predict.
Simple Model P(D|M) Complex Model P(D|M)
▲ ▲
│ █ │
│ █ │ ▄ █ ▄ █ ▄ ░ ░
│ █ │ █ █ █ █ █ █ █
└───────────► └───────────────►
Few Datasets Many Datasets
- A simple model has tight constraints and few parameters. It can only generate or explain a small, concentrated set of datasets. If the observed data happens to fall within this narrow window, the model's probability density for that data will be exceptionally high.
- A complex model has high flexibility and many degrees of freedom. It can adapt to explain a vast array of different datasets. However, because its total probability mass is conserved, it must spread its bets thinly across a much larger space of potential outcomes. Consequently, its probability for any single dataset is relatively low.
If both a simple model and a complex model manage to explain the observed data, the marginal likelihood will naturally favor the simple one. As David J. C. MacKay demonstrated in Bayesian Model Comparison and Backprop Nets, this evidence calculation inherently implements Occam's Razor without requiring manual tuning or artificial complexity penalties.
The Occam Factor in Action
When we integrate out the parameters $\theta$, we are essentially averaging the likelihood over our prior distribution. A complex model with a large parameter space requires a diffuse prior. Unless the extra parameters provide a massive boost to the likelihood over a significant portion of the parameter space, the process of integrating over a wide, unsupportive prior will dilute the overall evidence.
This mechanism is widely used across the physical sciences to prevent over-engineering. For instance, in cosmology, A. Lasenby et al. in Bayesian inflationary reconstructions from Planck 2018 data used nested sampling to compute this evidence, allowing them to establish a strict threshold for model complexity when reconstructing the early universe. Similarly, Edward Higson et al. showed in Bayesian sparse reconstruction: a brute-force approach to astronomical imaging and machine learning that the marginal likelihood can automate sparse signal reconstruction by naturally penalizing excessive parameterization.
Asymptotic Scaling via the BIC
Computing the full marginal likelihood requires integrating over high-dimensional parameter spaces, which can be computationally prohibitive. To circumvent this, researchers often use the Bayesian Information Criterion (BIC), an asymptotic approximation of the log marginal likelihood:
$$\text{BIC} = \ln(n)k - 2\ln(\hat{L})$$
where $n$ is the number of data points, $k$ is the number of parameters, and $\hat{L}$ is the maximized likelihood.
The BIC makes the complexity penalty explicit through the $\ln(n)k$ term. It provides a scalable way to navigate structural searches. For example, David Duvenaud et al. leveraged the BIC in Structure Discovery in Nonparametric Regression through Compositional Kernel Search to automatically discover interpretable Gaussian process kernels without overfitting. In discrete settings, Yutong Ye used the BIC as a stopping rule in Revisiting EM-based Estimation for Locally Differentially Private Protocols to prune noisy mixture components during expectation-maximization.
When Occam's Razor Cuts Too Deep
While the automatic penalty is powerful, it is not without failure modes. A key risk is underfitting. Because the marginal likelihood penalizes models with diffuse priors or high flexibility, it can sometimes reject highly expressive architectures that are actually capable of superior generalization.
Sanae Lotfi et al. explored this limitation in Bayesian Model Selection, the Marginal Likelihood, and Generalization, demonstrating that the marginal likelihood can exhibit an underfitting bias by over-penalizing complex models whose priors are spread too thin. This highlights a subtle misalignment: the marginal likelihood measures how well a model explains the data on average under its prior, which does not always perfectly correlate with its ability to predict unseen data points after training.