Imagine you have two normal distributions. Both have a mean of 0. The first has a standard deviation of 1, while the second has a standard deviation of 0.01. If you shift the mean of both distributions by a tiny increment of 0.1, what happens?
For the first distribution, the shift is barely noticeable; the two probability curves overlap almost completely. For the second distribution, however, that same 0.1 shift moves it entirely out of its original range, creating a completely distinct, easily distinguishable distribution.
This simple thought experiment reveals a fundamental flaw in using standard Euclidean distance to measure changes in probability distributions: a step of size $\Delta \theta$ in parameter space does not mean the same thing everywhere. The true "distance" between two distributions depends on how distinguishable they are based on data. This insight is the foundation of information geometry, which treats parametric families of probability distributions as curved Riemannian manifolds rather than flat Euclidean spaces.
The Fisher Information Matrix as a Metric Tensor
To measure distances on a curved surface, we need a metric tensor—a mathematical tool that defines a local "ruler" at every point. In information geometry, this ruler is the Fisher Information Matrix (FIM). As Frank Nielsen explains in The Many Faces of Information Geometry, the FIM is the unique Riemannian metric that remains invariant under transformations of data using sufficient statistics.
By treating the FIM as a metric tensor, we warp the parameter space so that distances correspond directly to statistical discriminability. In regions of high variance, the space is compressed because distributions are hard to tell apart; in regions of low variance, the space is stretched. As noted by Benczur et al., normalizing representations using the FIM effectively transforms gradient vectors into a coordinate system on a Riemannian manifold, ensuring that distances reflect true statistical differences.
Geodesics and the Fisher-Rao Distance
If the FIM gives us a local ruler, how do we measure the distance between two far-apart distributions? We find the shortest path along the curved manifold—the geodesic. The length of this path is known as the Fisher-Rao distance.
Using the Fisher-Rao distance instead of Euclidean or Mahalanobis metrics provides significant advantages in practical machine learning tasks:
- Out-of-Distribution (OOD) Detection: Standard metrics often fail when a test sample shares a mean with the training data but differs wildly in variance. Gomes et al. demonstrated that replacing Euclidean metrics with the Fisher-Rao distance allows detectors to account for the underlying geometric curvature, dramatically improving OOD detection across varying model access scenarios.
- Signal Processing and Sparse Recovery: In "off-the-grid" sparse recovery (such as decomposing mixtures of spikes), Euclidean distance fails to capture the relationships between overlapping features. Poon et al. proved that the Fisher-Rao distance serves as the natural metric for defining spike separation, providing the first rigorous support recovery guarantees for non-translation-invariant problems.
The Power of Coordinate Invariance
A major headache in machine learning is reparameterization. If you change how you describe a model (for example, switching from variance to precision, or rescaling weights), your optimization landscape changes. However, because information geometry operates on the space of distributions rather than the arbitrary coordinates of their parameters, its geometric properties are invariant under smooth, invertible transformations (Ketema et al.).
This invariance can be exploited to solve long-standing problems in deep learning generalization. For instance, researchers have long known that "flat" regions in a loss landscape correlate with good generalization, but standard sharpness measures can be fooled by simply rescaling a network's weights. To fix this, Jang et al. introduced an Information Geometric Sharpness (IGS) measure. Because IGS measures sharpness on the Fisher information manifold rather than the Euclidean parameter space, it remains perfectly invariant under reparameterization and serves as a highly robust training regularizer.
Computational Strategies: Isometries and Proxies
Computing exact geodesics on a statistical manifold can be analytically intractable. Researchers handle this challenge using three primary strategies:
- Automatic Differentiation: Tools like the
Geomstatslibrary developed by Le Brigant et al. leverage automatic differentiation backends to compute the Fisher-Rao metric for arbitrary user-defined density functions, bypassing the need for manually derived closed-form solutions. - Geometric Isometries: Sometimes, a statistical manifold is perfectly isometric to a well-studied geometric space. For example, Beinert and Bresch utilize the isometry between the Fisher metric of univariate Gaussians and hyperbolic space to transform a non-convex statistical denoising problem into a manageable convex optimization problem. Similarly, Chakraborty et al. map probability distributions onto a unit hypersphere to simplify the aggregation of weak classifiers.
- Tractable Proxies: When the exact Fisher-Rao metric is too expensive to compute, symmetric $f$-divergences like the Hellinger distance can serve as computationally efficient proxies that still preserve the underlying Riemannian geometry (Baktashmotlagh et al.; Mathieu Salzmann).
Navigating Singularities
What happens when the Fisher Information Matrix becomes non-invertible or degenerate? This is a common issue in overparameterized models like deep neural networks or mixture models, where multiple parameter combinations yield the exact same output distribution.
Kenji Fukumizu addressed this breakdown of classical statistical theory by reformulating likelihood ratio tests using tangent cones to handle non-invertible regions smoothly. Taking this further to explain the mystery of deep learning generalization, Sun and Nielsen extended information geometry to singular semi-Riemannian manifolds. They showed that directions in parameter space with zero eigenvalues act as "lightlike dimensions" that do not alter the model's output distribution. This framework provides a geometric explanation for Occam's razor in deep learning, proving that the singularities in massive networks actually reduce their effective complexity, allowing them to generalize well despite having millions of parameters.
Key Foundational Work
A Theory of Mean Field Approximation (1998) — contributed 4 concept elements to this direction.
Counting probability distributions: Differential geometry and model selection (2000) — contributed 2 concept elements to this direction.