Abstract
In Bayesian inference and statistical physics, Annealed Importance Sampling (AIS) is a cornerstone for sampling from complex distributions by constructing intermediate annealing paths. This paper provides a rigorous geometric and variational justification for these paths using the rho-tau ($ρ$-τ) Bregman divergence framework. The authors demonstrate that common paths, such as geometric and $q$-paths, are unique minimizers of a generalized Bregman Information objective and represent geodesics within specific information-geometric manifolds. This work bridges the gap between ad-hoc interpolation methods and formal information geometry, particularly for the unnormalized densities prevalent in MCMC.
Introduction
In the world of Bayesian inference, sampling from "difficult" distributions often requires Annealed Importance Sampling (AIS). To transition from a simple distribution (e.g., a Gaussian) to a complex posterior, researchers construct an annealing path—a sequence of intermediate distributions that bridge the gap. While geometric averaging is the standard choice, recent developments have introduced $q$-paths based on non-extensive thermodynamics.
Despite their utility, these paths often appear as ad-hoc choices. This paper, "Variational Representations of Annealing Paths: Bregman Information under Monotonic Embedding," provides a deep geometric justification, showing these paths are optimal solutions within a specific information-geometric framework.
The Problem: Ad-Hoc Annealing Paths
When performing MCMC sampling, we move along a path $\tilde{\pi}_\beta$ where $\beta \in [0, 1]$. While arithmetic means minimize expected Euclidean distance, it was unclear if a similar optimality existed for the complex paths used in MCMC, especially for unnormalized densities. Previous work addressed specific cases like KL divergence, but a unified theory connecting different means, divergences, and the underlying geometry was missing.
Theoretical Framework: Rho-Tau Bregman Divergence
The authors leverage the rho-tau ($ρ$-τ) Bregman divergence framework to generalize standard information geometry. This approach relies on two key components:
- Quasi-Arithmetic Means: Instead of simple weighted sums, the authors consider means of the form $\mu_\rho = \rho^{-1}((1-\beta)\rho(\pi_0) + \beta\rho(\pi_1))$. Different "embedding" functions $\rho$ (such as the $q$-logarithm) recover different path types.
- Variational Representation: Theorem 1 proves that these quasi-arithmetic means are the unique minimizers of an "expected rho-tau Bregman divergence," termed the Rho-Tau Bregman Information.
Key Results: Optimality and Geodesics
The paper establishes two primary theoretical results:
- Theorem 1 (Rho-Tau Bregman Information): Formally establishes that the quasi-arithmetic mixture minimizes the expected divergence, generalizing results from standard Bregman divergences to a wider class of embeddings.
- Theorem 2 (Geodesics): Confirms that these paths are auto-parallel (geodesics) with respect to the affine connections induced by these divergences. In essence, these paths are the "straightest" possible routes between two distributions in a curved space.
Furthermore, the framework explicitly handles unnormalized measures ($\tilde{\pi}$), which is critical for MCMC where normalization constants are typically unknown.
Insights and Implications
- The "Barycenter" Connection: Annealing paths are essentially moving barycenters (generalized centers of mass). Changing the mixing weight $\beta$ finds the point "closest" to the start and end distributions.
- Duality: The framework relies on a "referential-representational biduality." Choosing how you represent the density ($ρ$) and measure change ($τ$) defines the entire geometry of the sampling path.
- Parametric Unification: These paths can be viewed as one-dimensional deformed exponential families, linking non-parametric density spaces to parametric models.
Conclusion and Future Directions
While providing an elegant theoretical foundation, the authors acknowledge practical challenges, such as the computational intractability of some complex embeddings. Future work may involve learning optimal paths adaptively or applying this framework to Variational Inference (VI) and Reinforcement Learning (RL). By viewing annealing paths as geodesics, this research provides a rigorous "straight line" justification for efficient sampling strategies.
Summary updated