Predictive Inference via Bayesian Model Averaging

346 concepts241 papers6 proposals

When building predictive models, the standard operating procedure is to find the "best" model and discard the rest. Whether we are selecting a sparse set of features via stepwise regression, tuning hyperparameters, or training a deep neural network to find an optimal point estimate of weights, we treat model selection as a winner-take-all contest.

This approach suffers from a fundamental flaw: model uncertainty. Often, many distinct models or parameter configurations can explain the training data equally well, yet they diverge significantly when making predictions on unseen data. Committing to a single point estimate ignores this underspecification, leading to overconfident and brittle predictions.

As Andrew Gordon Wilson argues in The Case for Bayesian Deep Learning, accounting for this inherent underspecification is crucial for robust generalization. Bayesian Model Averaging (BMA) offers a mathematically principled alternative to single-model selection. Instead of forcing a choice, BMA retains a collection of candidate models and averages their predictions, weighting each by its posterior probability given the observed data.


The Core Mechanism: Marginalization over Model Space

To understand why Bayesian Model Averaging works, consider the task of predicting an outcome $y$ given an input $x$ and a dataset $D$. In a traditional pipeline, we select a single optimal model $\hat{m}$ and compute $P(y|x, \hat{m})$.

BMA replaces this optimization step with integration (or summation). By applying the law of total probability, the predictive distribution is framed as:

$$P(y|x, D) = \sum_{m \in \mathcal{M}} P(y|x, m, D) P(m|D)$$

Here, $\mathcal{M}$ represents the space of all candidate models. The prediction is a weighted average where the weights, $P(m|D)$, are the posterior probabilities of each model. According to Bayes' theorem, this weight is proportional to the model's prior probability $P(m)$ and its marginal likelihood (or evidence) $P(D|m)$:

$$P(m|D) \propto P(D|m) P(m)$$

The marginal likelihood automatically penalizes overly complex models, acting as a built-in Occam's razor. A model that is too flexible spreads its probability mass too thinly over the data space, resulting in a lower marginal likelihood than a simpler model that fits the data well.

As Eduardo Ley demonstrates in Bayesian Modelling of Catch in a North-West Atlantic Fishery, relying on a single model chosen via stepwise selection frequently yields overconfident predictions. By treating the selection of variables as an unknown parameter and averaging over a vast model space, BMA explicitly captures structural uncertainty, leading to more stable and calibrated predictive intervals. This stabilization is particularly pronounced when datasets are small or when multiple distinct hypotheses are empirically plausible, as noted by Sri Astuti Thamrin et al. in Modelling survival data to account for model uncertainty: a single model or model averaging?.


From Discrete Models to Continuous Parameter Spaces

While BMA was historically applied to discrete sets of statistical models (such as choosing which variables to include in a linear regression), the same underlying intuition applies to continuous parameter spaces in machine learning.

In deep learning, finding a single "optimal" weight configuration via Stochastic Gradient Descent (SGD) is analogous to selecting a single model. However, neural network loss landscapes contain vast valleys of distinct weight settings that achieve identical training errors but generalize differently.

Instead of relying on a single point estimate, we can marginalize over the weight distribution. For instance, Carlo Lucibello et al. show in Deep learning via message passing algorithms based on belief propagation that utilizing the full distribution of weights provided by message-passing marginals yields superior accuracy and better uncertainty quantification compared to point-estimate methods.

Point Estimate (Standard ML):
Input (x) ───> [ Single "Best" Model / Weights ] ───> Overconfident Prediction (y)

Bayesian Model Averaging:
               ┌───> Model A (Weight: 0.50) ───┐
Input (x) ────┼───> Model B (Weight: 0.35) ───┼───> Blended, Calibrated Prediction (y)
               └───> Model C (Weight: 0.15) ───┘

When explicit integration over weights is computationally intractable, researchers use clever approximations to achieve the benefits of BMA:

  1. Precision-Weighted Merging: Michael Matena et al. introduce a technique called Fisher Merging in Merging Models with Fisher-Weighted Averaging. By approximating the posterior distributions of different pre-trained models as Gaussians, they frame model merging as a Bayesian inference problem. The merged model's parameters become a precision-weighted combination of the individual models' weights, using the Fisher Information matrix to determine which parameters are critical to preserve.
  2. Subspace and Trajectory Ensembling: Rather than treating independent training runs as isolated solutions, methods can combine frequentist ensembling with Bayesian posterior estimation. As explored in Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation, integrating multiple independent training trajectories with Gaussian approximations of the weight space provides robust epistemic uncertainty quantification.

Averaging vs. Combination: What if the True Model is Missing?

A subtle but vital theoretical distinction exists between Bayesian Model Averaging (BMA) and Bayesian Model Combination (BMC).

BMA operates under the " $\mathcal{M}$-closed" assumption—it assumes that the true data-generating process is exactly one of the models in the candidate set $\mathcal{M}$. As more data is collected, BMA typically behaves as a selection mechanism, asymptotically assigning a weight of $1.0$ to the single model that is closest to the truth in a Kullback-Leibler sense, while driving the weights of all other models to $0.0$.

However, in real-world scenarios, our candidate models are merely approximations; the true data-generating process is rarely contained within our set (the " $\mathcal{M}$-open" view). In these settings, forcing models to compete for a single crown can be suboptimal.

Bart van Erp et al. address this in Automating Model Comparison in Factor Graphs. They note that BMC demonstrates superior performance over BMA when the true data-generating process is absent from the candidate set. Instead of trying to identify the single "true" model, BMC optimizes a weighted ensemble of models to approximate the complex data-generating distribution, allowing multiple models to permanently coexist and complement one another.


Modern Applications: From Spatial Fields to Large Language Models

The principle of marginalizing over model structures has found success across diverse domains, demonstrating its versatility as a foundational machine learning paradigm.

Domain Adaptation and Fusion

When adapting pre-trained models to a new target domain without accessing their original training data, selecting the single "closest" source model can fail if the target domain contains mixed characteristics. Zhongyi Han et al. address this in Discriminability and Transferability Estimation: A Bayesian Source Importance Estimation Approach for Multi-Source-Free Domain Adaptation by building a Bayesian framework where model discriminability serves as the prior and transferability acts as the likelihood. This allows for a principled, weighted combination of source models tailored to the new task. Similarly, in complex spatial tracking environments, Peng Wu et al. leverage a Bayesian fusion strategy in Personalized Federated Learning over non-IID Data for Indoor Localization to probabilistically blend predictions from multiple localized models, resolving ambiguity when the exact context of an input is unknown.

Automating Probabilistic Programming

BMA is also being used to automate statistical modeling itself. In Large Language Bayes, Justin Domke utilizes a Large Language Model to translate informal text descriptions into a suite of candidate probabilistic programs. Rather than attempting to pick the single best program generation, the framework uses BMA to integrate predictions across all generated programs, weighting each by its marginal likelihood relative to the observed data. This ensures that programs empirically supported by the data dominate the final prediction, even if the LLM generated syntactically plausible alternatives.

By shifting the goal from finding the single "correct" model to integrating across a distribution of plausible models, Bayesian Model Averaging and its variants provide a mathematical framework for honest uncertainty quantification and robust predictive performance.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 247 papers
  2. Version 1 · 2026-10-04 · 199 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_bayesian_model_averaging_17657310,
  title = {Bayesian model averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-model-averaging/rdr_176573105ba845b492be8fb89e046c04},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_bayesian_model_averaging_17657310_v2,
  title = {Bayesian model averaging},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bayesian-model-averaging/rdr_176573105ba845b492be8fb89e046c04/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake