A Theory of Mean Field Approximation

Toshiyuki TanakaNeurIPS 1998 · OpenReview

Overview

This paper provides a unifying theoretical framework for mean field approximations—ranging from the "naive" version to the advanced TAP approach—using the lens of information geometry. By treating probability distributions as points on a geometric manifold, the author demonstrates that these seemingly distinct approximations are essentially different orders of a Taylor expansion (the Plefka expansion) of the system's free energy.

The Problem: Computational Limits in Statistical Physics

In the realm of neural networks and statistical physics, Boltzmann machines are powerful models but notoriously expensive to compute. Calculating the expected state of a neuron or the correlation between two neurons requires summing over an exponential number of possible states, which is practically impossible for large networks.

To circumvent this, researchers rely on Mean Field Approximation (MFA). MFA simplifies the problem by assuming neurons act somewhat independently or are influenced only by an "average" field from their neighbors. Historically, MFA has come in several distinct "flavors":

  1. Naive MFA: The simplest version, often used in basic neural network models.
  2. TAP (Thouless-Anderson-Palmer) approach: A more accurate version derived from spin-glass theory that includes a "reaction term."
  3. Linear Response Theorem: A method used to estimate correlations between variables.

Before this paper, these methods were often treated as distinct tricks or heuristics. The author seeks to answer a fundamental question: Is there a single mathematical framework that explains and unifies all of them?

The Solution: Information Geometry

The author employs Information Geometry, a field that applies differential geometry to probability theory, to build this unified framework.

  • Statistical Manifolds: The set of all possible states of a Boltzmann machine is treated as a multidimensional surface (a manifold).
  • Dual Foliations: The author divides this manifold into "leaves" (sub-surfaces). One set of leaves ($\mathcal{F}$) represents distributions with fixed connection weights, while the other ($\mathcal{A}$) represents distributions with fixed average activations (means).
  • KL Divergence as Distance: The "closeness" of an approximation to the true distribution is measured using Kullback-Leibler (KL) divergence, which the author interprets geometrically as a distance on the manifold.
Information geometry of mean field approximation, showing the orthogonal projection between manifolds.

By utilizing a "Pythagorean law" for KL divergence, the paper explains why minimizing the distance to a simplified model (naive MFA) is a mathematically valid way to approximate the true distribution. The "naive" mean field approximation is shown to be a geometric projection of a complex distribution onto a simpler, factorizable sub-manifold.

Key Insights: The Plefka Expansion

The core mathematical technique used to connect these geometric concepts to practical equations is a Taylor expansion of the Gibbs free energy, known as the Plefka Expansion. By expanding this energy around a point where connections are zero (the "factorizable" model where neurons are independent), the author derives different levels of approximation.

  • The "Aha!" Moment on TAP: The author proves that the TAP approach is simply the second-order Taylor expansion of the free energy. The famous "Onsager reaction term" in physics emerges naturally as the second-order coefficient in this expansion.
  • Beyond "Large N": Traditionally, it was believed that MFA only works when the number of neurons ($N$) is very large. This framework argues that MFA's validity actually depends on whether higher-order terms in the Plefka expansion vanish, which can happen even in small networks.
Different orders of the Plefka expansion applied to a 2-unit Boltzmann machine toy model.

Using a 2-unit Boltzmann machine as a toy example, the author visualizes how increasing the order of the expansion (from 0th to 4th order) changes the approximation curves, bringing them closer to the true distribution dynamics.

Key Results

  • Unified Equations: The author successfully derives the standard mean field equations (1st order) and the TAP equations (2nd order) from the exact same expansion formula.
  • Geometric Interpretation of Linear Response: The Linear Response Theorem is shown to be an identity involving the Fisher Information Matrix, which acts as the metric tensor of the statistical manifold.
  • Higher-Order Potential: The paper provides explicit formulas for third-order coefficients, offering a theoretical path to create even more accurate "super-TAP" approximations.

Limitations and Future Work

While theoretically elegant, the framework has practical limitations.

  • Intractability of High Orders: While the framework allows for $n$-th order approximations, calculating coefficients beyond the third order becomes extremely complex because the sub-manifold $\mathcal{A}(m)$ does not belong to an "exponential family."
  • Scope: The derivation is specific to Boltzmann machines (binary variables). While extensions to wider statistical models are possible, they require further research.
  • Convergence: The paper leaves aside the "convergence problem"—whether the Taylor expansion actually settles on a stable value for all types of network architectures.

Summary updated

Abstract

I present a theory of mean field approximation based on information ge(cid:173) ometry. This theory includes in a consistent way the naive mean field approximation, as well as the TAP approach and the linear response the(cid:173) orem in statistical physics, giving clear information-theoretic interpreta(cid:173) tions to them.
BibTeX
@inproceedings{tanaka1998theory,
  author = {Toshiyuki Tanaka},
  title = {{A Theory of Mean Field Approximation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1998},
  url = {https://openreview.net/forum?id=SkEOFP-uZS},
}
Figure