Abstract
We investigate the approximation of high-dimensional target measures as low-dimensional updates of a dominating reference measure. This approximation class replaces the associated density with the composition of: (i) a feature map that identifies the leading principal components or features of the target measure, relative to the reference, and (ii) a low-dimensional profile function. When the reference measure satisfies a subspace $\phi$-Sobolev inequality, we construct a computationally tractable approximation that yields certifiable error guarantees with respect to the Amari $\alpha$-divergences. Our construction proceeds in two stages. First, for any feature map and any $\alpha$-divergence, we obtain an analytical expression for the optimal profile function. Second, for linear feature maps, the principal features are obtained from eigenvectors of a matrix involving gradients of the log-density. Neither step requires explicit access to normalizing constants. Notably, by leveraging the $\phi$-Sobolev inequalities, we demonstrate that these features universally certify approximation errors across the range of $\alpha$-divergences $\alpha \in (0,1]$. We then propose an application to Bayesian inverse problems and provide an analogous construction with approximation guarantees that hold in expectation over the data. We conclude with an extension of the proposed dimension reduction strategy to nonlinear feature maps.
Introduction
In fields ranging from Bayesian statistics to statistical mechanics, the challenge of high-dimensional probability measures is pervasive. Standard computational tools, such as Markov chain Monte Carlo (MCMC), often suffer from the "curse of dimensionality," where performance degrades exponentially as the number of variables increases. A common strategy to mitigate this is dimension reduction: identifying a low-dimensional subspace or "feature set" that captures the essential behavior of a complex target measure $\pi$ relative to a simpler reference measure $\eta$.
This paper introduces a unified mathematical framework for this compression. By utilizing $\phi$-Sobolev inequalities, the authors provide a method to identify principal features that offer certifiable error guarantees across a broad spectrum of statistical metrics, including Kullback–Leibler (KL) divergence and Hellinger distance.
The Approximation Framework
The proposed method approximates the target density by decomposing it into a feature map and a profile function. Specifically, the density is modeled as the composition of a map that extracts the most informative directions and a low-dimensional function that describes the density's behavior along those directions.
Amari $\alpha$-Divergences
To quantify the quality of these approximations, the framework employs Amari $\alpha$-divergences. This family of distance measures is parameterized by $\alpha$, encompassing several standard metrics:
- $\alpha = 1$: The Kullback–Leibler (KL) divergence.
- $\alpha = 1/2$: The squared Hellinger distance.
$\Phi$-Sobolev Inequalities
The theoretical backbone of the work is the use of $\phi$-Sobolev inequalities. These functional inequalities relate the entropy (or spread) of a function to its gradient (or "wiggliness"). The authors demonstrate that if the reference measure $\eta$ satisfies these inequalities, the approximation error can be bounded by the gradients of the target density. A key discovery is the conjugacy relation: dimension reduction for a divergence of order $\alpha$ is controlled by a Sobolev inequality of order $1/\alpha$.
Principal Feature Detection
The core algorithm for identifying these features involves the construction of a Diagnostic Matrix ($H$). This matrix is defined by integrating the outer product of the log-density gradients:
$$H = \int \nabla \log \ell(x) \nabla \log \ell(x)^\top d\pi(x)$$
The "principal features" are identified as the top $r$ eigenvectors of $H$. This spectral approach provides a computationally tractable way to find the most informative linear directions in the high-dimensional space.
Universal Robustness
A significant result of this research is the "universal" nature of the features derived from $H$. The same set of eigenvectors provides a certifiable bound on the error regardless of which specific $\alpha$-divergence (within the range $(0, 1]$) is used. This allows practitioners to perform dimension reduction once and remain confident in the approximation's accuracy across various statistical contexts.
Applications and Extensions
Bayesian Inverse Problems
The framework is particularly potent in Bayesian settings. The authors show that one can identify important features a priori—before observing data. By averaging the Fisher information matrix over the prior distribution, a subspace can be pre-calculated that is guaranteed to be effective on average for any future data realization.
Nonlinear Feature Maps
While linear projections are standard, the authors extend their theory to nonlinear feature maps using normalizing flows (diffeomorphisms). This allows the framework to handle complex, non-linear dependencies in the target distribution, enabling more aggressive and accurate dimension reduction for sophisticated models.
Conclusion and Limitations
This work provides a rigorous foundation for dimension reduction in high-dimensional probability measures, offering sharper bounds than previous methods for metrics like the Hellinger distance. However, some constraints remain:
- Reference Measure: The theory requires the reference measure to satisfy subspace $\phi$-Sobolev inequalities, which may exclude heavy-tailed distributions.
- The $\chi^2$ Gap: The current tools do not easily extend to $\alpha > 1$ (e.g., $\chi^2$-divergence).
- Computational Cost: Constructing the matrix $H$ requires integration over the target measure, which, while often more efficient than full sampling, still necessitates numerical approximation in high-dimensional settings.
Summary updated