Nonparametric Fisher Geometry with Application to Density Estimation

Andrew Holbrook, Shiwei Lan, Jeffrey Streets, Babak ShahbabaUAI 2020 · OpenReview

Abstract

It is well known that the Fisher information induces a Riemannian geometry on parametric families of probability density functions. Following recent work, we consider the nonparametric generalization of the Fisher geometry. The resulting nonparametric Fisher geometry is shown to be equivalent to a familiar, albeit infinite-dimensional, geometric object—the sphere. By shifting focus away from density functions and toward square-root density functions, one may calculate theoretical quantities of interest with ease. More importantly, the sphere of square-root densities is much more computationally tractable. As discussed here, this insight leads to a novel Bayesian nonparametric density estimation model.

Introduction

Density estimation is a fundamental task in statistics, yet it presents significant challenges when moving beyond rigid parametric models. Nonparametric methods offer the necessary flexibility to capture complex data distributions but must contend with two strict mathematical constraints: non-negativity ($p(x) \geq 0$) and the unit-integral constraint (the area under the curve must sum to exactly one). Traditional approaches, such as the Gaussian Process Density Sampler (GPDS), often struggle with these constraints, leading to high computational costs—typically $O(N^3)$—and complex sampling requirements.

This work addresses these challenges by generalizing Fisher information geometry to the nonparametric setting. The core insight is that the space of all probability densities, when viewed through the lens of Fisher geometry, is isometric to an infinite-dimensional unit sphere. This geometric perspective transforms the difficult task of sampling valid densities into the more manageable problem of sampling points on a high-dimensional ball.

Methodology

The proposed framework utilizes several key mathematical techniques to bridge the gap between theoretical geometry and practical computation:

  1. The Square-Root Transformation: Instead of modeling the density $p(x)$ directly, the model focuses on its square root, $q(x) = \sqrt{p(x)}$. This transformation naturally satisfies the non-negativity constraint.
  2. Fisher Geometry as a Sphere: The space of square-root densities is shown to be a unit sphere in an infinite-dimensional Hilbert space ($L^2$).
  3. Karhunen-Loève (K-L) Expansion: To handle the infinite-dimensional nature of the sphere, the square-root density is represented via a weighted sum of orthonormal basis functions (e.g., Laplacian eigenfunctions). This reduces the functional estimation problem to estimating a finite set of coefficients.
  4. Spherical Hamiltonian Monte Carlo (HMC): Since the coefficients are constrained to a sphere, the authors employ Spherical HMC. This algorithm efficiently explores the posterior by following geodesics (shortest paths) on the sphere.
  5. The $\chi^2$-process Prior: By placing a Gaussian Process prior on the square-root density, the model induces a $\chi^2$-process prior on the resulting density, providing a flexible Bayesian framework.
The model demonstrates high flexibility in 1D density estimation across various distribution shapes, including uniform, unimodal, and U-shaped distributions.

Experiments and Results

The model was evaluated on synthetic 1D and 2D datasets, as well as real-world data including British Coal Mine disasters and Bramble Canes spatial locations.

A major advantage of this approach is its computational efficiency. Unlike standard Gaussian Process methods that scale cubically, the basis-expansion approach scales linearly ($O(N)$) with the number of data points. This allows the model to handle large datasets (1,000+ points) with ease while maintaining smooth posterior draws.

In 2D experiments, the model successfully recovered complex, multimodal shapes. For instance, it accurately captured the underlying structure of a noisy "circle" distribution, a task that typically challenges simpler parametric models.

2D density estimation of a complex, noisy circular pattern, demonstrating the model's ability to recover multimodal structures.

Furthermore, the $\chi^2$-process prior simplifies the likelihood calculations for Cox processes, which are frequently used in spatial statistics. By avoiding the difficult integrals usually associated with these models, the proposed method offers a more streamlined path for spatial density estimation.

Conclusion

This paper provides a rigorous theoretical foundation for nonparametric density estimation by leveraging the isometry between Fisher geometry and the $L^2$ sphere. While the choice of basis functions and the truncation of the K-L expansion remain important considerations, the resulting model is both theoretically elegant and computationally efficient. Future work may involve placing priors on the number of basis functions to allow the model to automatically adjust its complexity based on the data.

Summary updated

Abstract

It is well known that the Fisher information induces a Riemannian geometry on parametric families of probability density functions. Following recent work, we consider the nonparametric generalization of the Fisher geometry. The resulting nonparametric Fisher geometry is shown to be equivalent to a familiar, albeit infinite-dimensional, geometric object—the sphere. By shifting focus away from density functions and toward square-root density functions, one may calculate theoretical quantities of interest with ease. More importantly, the sphere of square-root densities is much more computationally tractable. As discussed here, this insight leads to a novel Bayesian nonparametric density estimation model.
BibTeX
@inproceedings{holbrook2020nonparametric,
  author = {Andrew Holbrook and Shiwei Lan and Jeffrey Streets and Babak Shahbaba},
  title = {{Nonparametric Fisher Geometry with Application to Density Estimation}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=q5kxGneV23},
}
Figure