On quantitative Laplace-type convergence results for some exponential probability measures, with two applications

Valentin De Bortoli, Agnès DesolneuxarXiv 2021 · arXiv

Abstract

Classical Laplace approximations provide a framework for understanding the concentration of probability measures as a temperature parameter $\epsilon$ approaches zero. However, these methods typically rely on the invertibility of the potential function's Hessian at its minimum. In "On quantitative Laplace-type convergence results for some exponential probability measures, with two applications," Valentin De Bortoli and Agnès Desolneux address the "degenerate" case where the set of minimizers forms a manifold rather than a single point. By employing tools from geometric measure theory, specifically the coarea formula, the authors establish non-asymptotic, quantitative convergence bounds in the Wasserstein distance, offering new insights into maximum entropy models and the low-temperature behavior of Stochastic Gradient Langevin Dynamics (SGLD).

Introduction

In probability theory and statistical physics, we frequently encounter "cold" distributions—measures that concentrate around the minima of a potential function $U$ as a temperature parameter $\epsilon$ vanishes. The classical Laplace method provides an asymptotic expansion for integrals of the form $\int \exp[-U(x)/\epsilon] dx$, characterizing the limiting distribution.

While robust in many settings, the standard theory possesses a significant limitation: it requires the Hessian $\nabla^2 U$ to be non-singular at the global minimum. In many modern applications—such as over-parameterized neural networks or constrained optimization—the minimum is not a discrete point but a manifold (e.g., a circle, a line, or a complex subspace). In these instances, the Hessian is singular, and the standard mathematical machinery fails to provide quantitative rates of convergence. De Bortoli and Desolneux bridge this gap by providing a geometric framework that handles these degeneracies explicitly.

The Problem of Degeneracy

Consider a measure $\pi_\epsilon \propto \exp[-U(x)/\epsilon]$. As $\epsilon \to 0$, the measure concentrates on the set $\mathcal{M} = \text{argmin } U$. If $U$ is "flat" along certain directions—for instance, if $U(x) = (\|x\|^2 - 1)^2$, where the minimizers form a unit sphere—the standard Gaussian approximation is invalid.

Previous attempts to resolve these singularities often relied on complex algebraic geometry (resolution of singularities) or were restricted to purely asymptotic results. This paper seeks to provide non-asymptotic bounds: specifically, how close is $\pi_\epsilon$ to its limit $\pi_0$ for a specific, small value of $\epsilon$? The authors focus on "norm-like" potentials of the form $U(x) = \|F(x)\|^k$, where the properties of the map $F$ determine the geometry of the concentration.

Methodology: The Coarea Formula

The central innovation of this work is the application of the coarea formula from geometric measure theory. This formula allows for the evaluation of an integral by "slicing" the domain along the level sets of a function. By integrating over these slices, the authors can decouple the behavior of the measure along the manifold of minimizers from the behavior in the transverse (orthogonal) directions.

Instead of relying on the Hessian, the authors utilize a generalized Jacobian $JF(x)$. This term acts as a weighting factor that accounts for the "volume" or "density" of the measure as it collapses onto the manifold. This leads to the discovery of the "twisted" limit: the limiting measure $\pi_0$ is not necessarily a uniform distribution over the minimizers, but rather a distribution weighted by the local geometry of the potential.

Key Applications

1. Maximum Entropy Models

The paper clarifies the relationship between macrocanonical models (where constraints are satisfied on average) and microcanonical models (where constraints are satisfied exactly). The authors show that as the temperature goes to zero, the macrocanonical model converges to a "twisted" version of the microcanonical model, providing a rigorous link between these two fundamental concepts in statistical mechanics.

2. Stochastic Gradient Langevin Dynamics (SGLD)

In the context of non-convex optimization, the authors apply their results to SGLD. They distinguish between two types of barriers in the optimization landscape:

  • Kinetic Barriers: The height of the potential energy required to escape a local minimum.
  • Thermodynamic Barriers: The relative volume or mass associated with different minima.

Their analysis demonstrates that the thermodynamic barrier is the critical factor for quantitative convergence in low-temperature regimes. They prove that SGLD iterates concentrate on the global minimizers of the population risk, providing explicit rates that depend on the number of samples and the temperature parameter.

Quantitative Results

The authors establish that the distance between the finite-temperature measure $\pi_\epsilon$ and the limiting measure $\pi_0$ follows a specific rate of convergence:

  • Convergence Rate: $O(\epsilon^{1/k})$, where $k$ is the power of the norm-like potential.
  • Wasserstein Distance: They provide explicit constants for the $W_1$ metric, such that $W_1(\pi_\epsilon, \pi_0) \le A \epsilon^{1/k}$.

These results were validated using synthetic numerical experiments, including the study of zeros in high-degree polynomials and the concentration of measures on two-dimensional elliptical manifolds.

Conclusion and Future Directions

This research provides a rigorous mathematical foundation for understanding how probability measures concentrate in complex, degenerate landscapes. By moving beyond the requirement of invertible Hessians, the authors offer a toolkit that is more applicable to modern high-dimensional statistics and machine learning.

Future work may address potentials with sub-polynomial growth at infinity or landscapes with vanishing thermodynamic barriers. Nonetheless, for researchers in Bayesian inference and non-convex optimization, this paper offers a vital quantitative perspective on why and how fast our algorithms find global minima in the presence of geometric degeneracy.

Summary updated

Abstract

Laplace-type results characterize the limit of sequence of measures $(π_\varepsilon)_{\varepsilon >0}$ with density w.r.t the Lebesgue measure $(\mathrm{d} π_\varepsilon / \mathrm{d} \mathrm{Leb})(x) \propto \exp[-U(x)/\varepsilon]$ when the temperature $\varepsilon>0$ converges to $0$. If a limiting distribution $π_0$ exists, it concentrates on the minimizers of the potential $U$. Classical results require the invertibility of the Hessian of $U$ in order to establish such asymptotics. In this work, we study the particular case of norm-like potentials $U$ and establish quantitative bounds between $π_\varepsilon$ and $π_0$ w.r.t. the Wasserstein distance of order $1$ under an invertibility condition of a generalized Jacobian. One key element of our proof is the use of geometric measure theory tools such as the coarea formula. We apply our results to the study of maximum entropy models (microcanonical/macrocanonical distributions) and to the convergence of the iterates of the Stochastic Gradient Langevin Dynamics (SGLD) algorithm at low temperatures for non-convex minimization.
BibTeX
@misc{bortoli2021quantitative,
  author = {Valentin De Bortoli and Agnès Desolneux},
  title = {{On quantitative Laplace-type convergence results for some exponential probability measures, with two applications}},
  year = {2021},
  eprint = {2110.12922},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.12922},
}
Figure