Asymptotic Normality of Complex Bayesian Posteriors

22 concepts14 papers1 proposal

In Bayesian statistics, the posterior distribution represents everything we know about a parameter after seeing the data. In frequentist statistics, parameters are fixed, and uncertainty comes from the hypothetical replication of experiments. For decades, the Bernstein-von Mises (BvM) theorem has served as the conceptual bridge between these two worlds. It states that under mild regularity conditions, as the sample size $n$ grows to infinity, the posterior distribution shifts, narrows, and eventually becomes a multivariate Gaussian distribution centered at the maximum likelihood estimator.

When the BvM theorem holds, Bayesian credible intervals act exactly like frequentist confidence intervals. This is a beautiful synergy: you can use intuitive Bayesian modeling and sampling machinery, yet enjoy rigorous frequentist guarantees.

However, classical BvM theory was built for simple, parametric models where the number of parameters is fixed and small. Modern machine learning relies on high-dimensional models, infinite-dimensional nonparametrics, optimization-based approximations like Variational Inference (VI), and models that are fundamentally misspecified. A major frontier in statistical theory focuses on a vital question: Why, and when, does this asymptotic normality survive in complex, modern architectures?


High Dimensions and Nonparametrics: Finding Order in Infinite Space

In classical settings, the Gaussian limit emerges because the log-likelihood surface behaves locally like a quadratic function—a consequence of the Central Limit Theorem operating on independent data points. But when a model is nonparametric (having infinitely many parameters) or high-dimensional, the prior distribution can retain a permanent influence, or the posterior can spread out too thinly to form a clean Gaussian shape.

Remarkably, recent theory shows that the BvM phenomenon often survives if we look at the right projections. For instance, Ray (2017) demonstrated that in Gaussian white noise settings, adaptive Bayesian posteriors can behave asymptotically like Gaussians even without prior knowledge of the underlying data's smoothness. This allows Bayesian credible sets to function as valid frequentist confidence intervals in complex, infinite-dimensional spaces.

The core intuition is that while the entire infinite-dimensional parameter space might be too complex to become normal, the specific low-dimensional questions we ask of the model—such as smooth functionals or linear projections—benefit from an averaging effect.

Consider two prominent examples in modern machine learning:

  • Tree Ensembles: Bayesian Additive Regression Trees (BART) are highly non-linear, discrete structures. Yet, Rocková (2020) proved that the posterior distributions for smooth functionals of BART concentrate into a normal distribution centered at an efficient estimator. The ensemble's complexity averages out, justifying frequentist hypothesis testing on top of tree models.
  • Deep Neural Networks: In deep Bayesian Neural Networks (BNNs) used for feature selection, Liu et al. (2021) established the BvM phenomenon for variable importance measures. As the sample size grows, the posterior for feature importance concentrates normally, allowing practitioners to use credible intervals to rigorously manage the statistical risks of multiple comparisons.

Even when the underlying error distribution is completely unknown and non-Gaussian, Lin et al. (2018) showed that sparse linear regression models can maintain posterior consistency and contraction rates by leveraging Misspecified Local Asymptotic Normality (LAN) conditions.


Variational Inference as M-Estimation

Variational Bayes (VB) replaces expensive Markov Chain Monte Carlo (MCMC) sampling with optimization. Instead of sampling from the true posterior $p(\theta|X)$, VB finds a simpler distribution $q(\theta)$ (like a product of independent Gaussians) that minimizes the Kullback-Leibler (KL) divergence to the posterior.

Practitioners have long known that VB underestimates variance because the KL divergence penalizes the approximation heavily if $q(\theta)$ places mass where $p(\theta|X)$ is zero. Given this systemic underestimation, how could a BvM theorem possibly apply?

The breakthrough lies in shifting perspective from the shape of the variational posterior to the behavior of its center. Blei et al. (2017) established a variational Bernstein-von Mises theorem, proving that as data size increases, the VB posterior converges to a normal distribution centered at the true parameter. Even when the chosen variational family is a massive simplification of the true posterior, the point estimates remain frequentist consistent and asymptotically normal.

True Posterior (Complex, Correlated) ──> Concentrates to a point
                                                 │
Variational Optimization (KL Minimization) ──────┼──> Variational BvM:
                                                 │    The simplified VI mass
Variational Approximation (Simplified Shape) ────> tracks the true center normally

To understand why this happens, Westling (2019) reframed variational inference objectives as profiled M-estimation problems. In frequentist theory, an M-estimator maximizes an empirical contrast function (like a log-likelihood or a loss function). By optimizing out the auxiliary variational parameters, the main model parameters are effectively guided by a profiled objective function. Because M-estimators naturally enjoy asymptotic normality under very general conditions, the variational point estimate inherits this frequentist regularity. The optimization landscape smooths out with large datasets, forcing the variational mean to track the true parameter with Gaussian-like errors.


Managing Misspecification with Power Posteriors

What happens when our model is wrong? In the real world, the true data-generating process rarely matches our statistical assumptions. Under model misspecification, standard Bayesian posteriors can behave erratically, concentrating too aggressively around suboptimal parameters or yielding poorly calibrated uncertainties.

To combat this, researchers use $\alpha$-posteriors (or power posteriors), where the likelihood function is raised to a fractional power $\alpha \in (0, 1)$:

$$p_\alpha(\theta | X) \propto p(\theta) \prod_{i=1}^n p(x_i | \theta)^\alpha$$

Tempering the likelihood smooths out the posterior surface, preventing it from over-indexing on the flaws of a misspecified model.

Avella Medina et al. (2022) extended the BvM theorem to these $\alpha$-posteriors and their variational approximations under explicit model misspecification. They proved that the limiting distribution remains multivariate normal, but its variance is elegantly rescaled by a factor of $\alpha$. This scaling provides a predictable knob for robustness.

Crucially, this tempering does not destroy the utility of the model's point estimates. Avella Medina et al. (2023) demonstrated that the moments of these power posteriors converge to those of a standard BvM distribution, and the power posterior mean remains asymptotically equivalent to the Maximum Likelihood Estimator (MLE). By downweighting the likelihood, you insulate the inference from misspecification artifacts while preserving the optimal frequentist convergence rates of classical estimators.


The Analytical Toolkit

Proving that a complex posterior converges to a Gaussian requires moving beyond classical Taylor expansions. The modern literature relies on several distinct mathematical frameworks to establish these guarantees:

  1. Analytical Measure Theory: To handle complex topological challenges like inverse problems and multimodal distributions, Lu et al. (2017) developed a framework for directly approximating probability measures on $\mathbb{R}^d$, providing a more analytical proof structure for BvM extensions.
  2. Stochastic Process Expansions: For flexible nonparametric priors like Normalized Random Measures with Independent Increments (NRMIs), Zhang et al. (2024) characterized the limiting behavior of the posterior process via an asymptotic expansion that maps the distribution to a shifted Brownian bridge.
  3. Stein's Method: While asymptotic theorems describe what happens at infinity, practical inference requires knowing how fast we get there. Daly et al. (2021) highlighted how Stein's method can be used to derive data-dependent variance bounds, opening the door for explicit, finite-sample convergence rates for the BvM theorem in intricate models.

Whether applied to robust density estimation using Hellinger distances (Wu et al., 2018), challenging binomial problems with unknown parameters (Schmidt-Hieber et al., 2021), or pooling information in likelihood-free inference (Drovandi et al., 2025), the modern expansion of the Bernstein-von Mises phenomenon ensures that our most sophisticated computational tools remain anchored to solid frequentist foundations.

Key Foundational Work

Gaussian Approximations for Probability Measures on Rd (2017) — contributed 3 concept elements to this direction.
Frequentist Consistency of Variational Bayes (2017) — contributed 2 concept elements to this direction.
Asymptotic Properties for Methods Combining the Minimum Hellinger Distance Estimate and the Bayesian Nonparametric Density Estimate (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_bernstein_von_mises_phenomenon_6fd90654,
  title = {Bernstein-von Mises phenomenon},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bernstein-von-mises-phenomenon/rdr_6fd9065483ba4cfe925c4b9b49064de3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_bernstein_von_mises_phenomenon_6fd90654_v1,
  title = {Bernstein-von Mises phenomenon},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/bernstein-von-mises-phenomenon/rdr_6fd9065483ba4cfe925c4b9b49064de3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake