In standard machine learning, we often treat model parameters or data points as vectors in a flat, Euclidean space. We measure the distance between two models by looking at the straight-line difference between their weights. However, probability distributions do not live in a flat world. A small change in the parameters of a distribution might lead to a massive change in its behavior in one region of space, while a large change elsewhere might be barely perceptible.
Information Geometry provides the mathematical language to treat families of probability distributions as curved surfaces, known as statistical manifolds. By using the Fisher Information Metric (FIM) as a "ruler," we can measure distances based on how much information actually changes, rather than how much the raw numbers shift.
The Intuition of Distinguishability
To understand why Euclidean distance fails, consider two Gaussian distributions. If both have a very small variance (they are "sharp"), a tiny shift in their means makes them almost entirely non-overlapping—they become easy to distinguish. If they have a very large variance (they are "flat"), that same shift in mean results in two distributions that are still nearly identical.
As Andras A Benczur, Bálint Daróczy, and Lajos Rónyai explain in Fisher kernels for image descriptors: a theoretical overview and experimental results, the parameter space is inherently non-Euclidean. The Fisher Information Metric acts as a metric tensor that accounts for this curvature, recognizing that distributions with higher variance are "closer" in the space of information than those with low variance.
The Fisher-Rao Distance as the Shortest Path
On a curved surface like a sphere, the shortest path between two points isn't a straight line through the air, but a "great circle" along the surface. In information geometry, these shortest paths are called geodesics, and the distance along them is the Fisher-Rao distance.
This metric has a unique property: it is parameterization-invariant. As Frank Nielsen notes in Approximation and bounding techniques for the Fisher-Rao distances, the distance between two distributions remains the same whether you describe them by their mean and variance or by some complex non-linear transformation. This makes the Fisher-Rao metric a "natural" way to compare models.
Optimal Trajectories in Generative Models
One of the most exciting applications of this geometry is in generative modeling, particularly diffusion and flow matching. Standard diffusion models often use empirical "noise schedules" to turn data into noise. However, these schedules are often sub-optimal.
Chaoran Cheng and colleagues, in Categorical Flow Matching on Statistical Manifolds, show that by treating categorical data as points on a curved manifold, we can follow the mathematically "shortest paths" (geodesics) to transform noise into data. This prevents numerical instability at the boundaries of the probability simplex—the "walls" where probabilities must sum to one.
Similarly, Leo Zhang proved in The Cosine Schedule is Fisher-Rao-Optimal for Masked Discrete Diffusion Models that the popular "cosine schedule" used in many state-of-the-art models is actually a geodesic under the Fisher-Rao metric. By moving at a constant rate through the "space of probabilities," the model ensures that every step of the denoising process contributes an equal amount of information.
Robustness and Sensitivity
Information geometry also allows us to identify where models are "fragile." In A Geometrical Approach to Finding Difficult Examples in Language, Shashwat Kumar and P. Thomas Fletcher use the eigenvalues of the Fisher Information Metric to find sentences where a model's prediction is highly sensitive to minor changes. If the "curvature" of the model's output is very high at a certain point, it indicates that the model is relying on "shortcuts" rather than robust understanding.
This geometric perspective extends to adversarial defense. Marine Picot and Pablo Piantanida developed FIRE (Adversarial Robustness Via Fisher-Rao Regularization), which penalizes the geodesic distance between a model's predictions for clean and attacked images. By treating the softmax output as a point on a curved manifold, they provide a more precise measure of stability than standard Euclidean or KL-divergence metrics.
Mapping the Space of Tasks
Finally, information geometry helps us understand the relationships between different learning problems. Pratik Chaudhari suggests in An Information-Geometric Distance on the Space of Tasks that we can measure the distance between two tasks (like "identifying dogs" vs. "identifying cats") by calculating the shortest path a model's weights must travel on the statistical manifold.
This "map" of tasks, also explored by Rahul Ramesh in A Picture of the Space of Typical Learnable Tasks, reveals that the manifold of learned models is remarkably low-dimensional. The distances between models on this manifold often mirror human semantic hierarchies, suggesting that the geometry of information captures something fundamental about the structure of the world we are trying to model.