Mean-field variational inference with the TAP free energy: Geometric and statistical properties in linear models

Michael Celentano, Zhou Fan, Licong Lin, Song MeiarXiv 2023 · DOI · arXiv

In high-dimensional Bayesian inference, where the number of features $p$ is comparable to the number of samples $n$, practitioners often face a significant trade-off between computational speed and statistical accuracy. Variational Inference (VI) is frequently the preferred method for speed, as it transforms complex integration problems into more manageable optimization tasks. However, in these high-dimensional regimes, standard "mean-field" VI (MF-VI) is known to be systematically overconfident, underestimating uncertainty and providing biased estimates.

A recent study by Michael Celentano, Zhou Fan, Licong Lin, and Song Mei provides a rigorous geometric and statistical foundation for a superior alternative: the TAP (Thouless-Anderson-Palmer) free energy. By integrating tools from statistical physics and high-dimensional probability, the authors demonstrate that it is possible to achieve both computational efficiency and perfectly calibrated statistical inference in linear models.

The High-Dimensional Challenge

Consider the classic linear regression problem: $y = X\beta + \epsilon$. In the "classical" regime where $n \gg p$, the posterior distribution is typically well-behaved and contracts around the true parameter values. However, in the high-dimensional regime ($n \approx p$), the posterior does not contract as strongly.

Most practitioners utilize MF-VI, which approximates the posterior as a product of independent distributions by maximizing the Evidence Lower Bound (ELBO). The failure of MF-VI in high dimensions stems from its inability to account for the subtle correlations created by the design matrix $X$. This leads to "overconfidence"—the model reports marginal variances that are significantly smaller than the true posterior variances, resulting in unreliable scientific conclusions.

The TAP Free Energy and the Onsager Correction

The authors analyze the TAP free energy, an approximation originally derived in the context of spin glasses in statistical physics. The TAP objective differs from the standard mean-field objective by the inclusion of an Onsager correction term. This term accounts for the "feedback" effect, where an estimate of one variable slightly influences others through the shared data in the design matrix.

While the TAP objective is non-convex and has historically been viewed as difficult to optimize, this research investigates whether it possesses a "good" local minimizer that is both statistically consistent and computationally accessible.

Theoretical and Algorithmic Framework

The researchers' approach is divided into three primary components:

  1. Probabilistic Analysis: Using Gordon’s comparison inequality, the authors prove that a local minimizer exists in close proximity to the true posterior marginals. This establishes that the TAP objective, despite its complexity, contains the "correct" statistical information.
  2. Geometric Analysis: They demonstrate that while the TAP energy is not globally convex, it is locally strongly convex within a significant neighborhood of the true posterior. This is a crucial finding, as it suggests that standard optimization techniques can be effective once the algorithm is initialized correctly.
  3. Algorithmic Design: The paper proposes a two-stage optimization strategy:
  4. Stage 1: Utilize Approximate Message Passing (AMP) for a few iterations to enter the "basin of attraction" of the good local minimizer.
  5. Stage 2: Apply Natural Gradient Descent (NGD) to achieve linear convergence to the exact local minimizer.

Key Insights and Results

The study validates its theoretical findings through extensive numerical simulations using synthetic data with various priors (e.g., three-point distributions and Bernoulli-Gaussian sparse priors) and Gaussian design matrices.

Statistical Calibration

A standout result is the achievement of perfect calibration. When plotting Posterior Inclusion Probabilities (PIPs), the TAP estimates align almost perfectly with the $y=x$ line. In contrast, standard MF-VI estimates are wildly inaccurate, often claiming 100% certainty for incorrect variables. This suggests that even in "hard" regimes where the signal-to-noise ratio is low, the local minimizer found by the TAP method remains a reliable tool for inference.

Efficiency and Robustness

The TAP estimator consistently achieved lower Mean Squared Error (MSE) than standard MF-VI across various $n/p$ ratios. Furthermore, the authors tested the "universality" of their findings by using non-Gaussian design matrices (e.g., Rademacher or Bernoulli entries). The TAP method proved remarkably robust, suggesting that the benefits of the Onsager correction extend beyond the idealized Gaussian assumptions used in the proofs.

Limitations and Future Directions

While this work represents a breakthrough in making the TAP approach rigorous for linear models, several avenues for future research remain:

  • Formal Universality Proofs: While numerical evidence supports the use of non-Gaussian designs, a formal proof for these cases remains an open challenge.
  • Structured Priors: The current analysis assumes a product prior (independent coordinates) for $\beta$. Extending the framework to handle correlated or structured priors would increase its applicability to complex real-world datasets.
  • Generalized Linear Models (GLMs): The authors suggest that their "Gordon post-AMP" proof technique could be adapted for GLMs, such as logistic regression, which would significantly broaden the impact of TAP-based variational inference in the machine learning community.

By providing a clear path toward calibrated, high-dimensional inference, this research bridges the gap between the theoretical insights of statistical physics and the practical needs of modern Bayesian statistics.

Summary updated

Abstract

We study mean-field variational inference in a Bayesian linear model when the sample size n is comparable to the dimension p. In high dimensions, the common approach of minimizing a Kullback-Leibler divergence from the posterior distribution, or maximizing an evidence lower bound, may deviate from the true posterior mean and underestimate posterior uncertainty. We study instead minimization of the TAP free energy, showing in a high-dimensional asymptotic framework that it has a local minimizer which provides a consistent estimate of the posterior marginals and may be used for correctly calibrated posterior inference. Geometrically, we show that the landscape of the TAP free energy is strongly convex in an extensive neighborhood of this local minimizer, which under certain general conditions can be found by an Approximate Message Passing (AMP) algorithm. We then exhibit an efficient algorithm that linearly converges to the minimizer within this local neighborhood. In settings where it is conjectured that no efficient algorithm can find this local neighborhood, we prove analogous geometric properties for a local minimizer of the TAP free energy reachable by AMP, and show that posterior inference based on this minimizer remains correctly calibrated.
BibTeX
@misc{celentano2023mean,
  author = {Michael Celentano and Zhou Fan and Licong Lin and Song Mei},
  title = {{Mean-field variational inference with the TAP free energy: Geometric and statistical properties in linear models}},
  year = {2023},
  eprint = {2311.08442},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2311.08442},
}
Figure