The Information Geometry of Singular Learning Machines

This direction is split into other directions: Fisher information singularity. Its last version stays readable and citable here.

Version 1 of 175 papers108 conceptsfrozen 2026-10-04

In classical statistics, we often assume that models are "regular." This means that every set of parameters corresponds to a unique probability distribution and that the loss landscape looks like a tidy, quadratic bowl near the optimum. In this world, the Fisher Information Matrix (FIM)—which measures the curvature of the log-likelihood—is positive definite and invertible.

However, modern architectures like neural networks and mixture models are fundamentally singular. As Sumio Watanabe rigorously established in Algebraic Information Geometry for Learning Machines with Singularities, these models contain regions where the FIM becomes degenerate. This isn't just a mathematical technicality; it is the reason why traditional tools like the Bayesian Information Criterion (BIC) fail to predict how deep networks learn and generalize.

The Geometry of Redundancy

Why does the Fisher Information Matrix become singular? The culprit is usually redundancy. In a multilayer perceptron, if two hidden units have the same weights, or if a weight is set to zero, the model's output remains the same even if you change those parameters in certain directions.

Kenji Fukumizu demonstrated in Active Learning in Multilayer Perceptrons that these "mathematical collapses" prevent algorithms from calculating optimal data sampling because they rely on inverting the FIM. When the matrix is singular, the "curvature" in those redundant directions is zero—the landscape is perfectly flat.

Shun-ichi Amari and colleagues further characterized these regions as geometric cones in Geometrical Singularities in the Neuromanifold of Multilayer Perceptrons. Because of symmetries (like swapping two neurons), the parameter space isn't a smooth manifold but a collection of intersecting sheets. At these intersections, the standard "quadratic bowl" approximation of the loss function vanishes.

Why Singularities Cause Optimization Plateaus

If you have ever watched a neural network's training loss stall for thousands of iterations before suddenly dropping, you have likely encountered a singularity.

In Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition, Weili Guo and others show that when the FIM becomes singular, gradient-based optimization loses its directional guidance. The algorithm "wanders" in these flat regions because the gradient provides no information on which way to go.

This creates a paradox for second-order methods like Natural Gradient Descent (NGD). NGD tries to navigate the landscape by inverting the FIM to account for its curvature. But as Wu Lin and Frank Nielsen note in Tractable structured natural-gradient descent using local parameterizations, this inversion becomes numerically unstable or impossible at singular points.

The Real Log Canonical Threshold

If parameter count isn't a reliable measure of complexity for singular models, what is? Susan Wei and Hui Li argue in Deep Learning Is Singular, and That's Good that we must use Singular Learning Theory (SLT).

The key metric in SLT is the Real Log Canonical Threshold (RLCT). While classical theory says a model's complexity is roughly half the number of its parameters ($d/2$), the RLCT provides a "singular-aware" complexity measure. In overparameterized models, the RLCT is often much smaller than the parameter count.

This explains the "Occam's Razor" of deep learning: massive models generalize well because their effective complexity (the RLCT) is low. As Edmund Lau and Daniel Murfet describe in The Local Learning Coefficient: A Singularity-Aware Complexity Measure, the RLCT measures how "pointy" or "flat" a singularity is. "Flatter" singularities have smaller RLCTs, meaning they are simpler and more likely to generalize.

Generalization and Double Descent

The singular nature of these models also provides a geometric explanation for Double Descent, where increasing model size beyond the point of perfect training fit actually improves test performance.

Mahito Sugiyama and Prasad Cheema show in A Geometric Look at Double Descent Risk that as dimensionality increases, the "model volume" can actually shrink or saturate due to these singularities. Similarly, Ke Sun and Frank Nielsen propose in A geometric modeling of Occam’s razor in deep learning that singularities provide a "negative complexity" that counteracts the explosion of parameters.

Summary of the Shift

Feature Regular Statistical Theory Singular Learning Theory
Fisher Information Positive definite, invertible Singular, non-invertible
Loss Landscape Quadratic bowl Cones, flat valleys, intersections
Complexity Metric Parameter count ($d/2$) Real Log Canonical Threshold (RLCT)
Asymptotics Gaussian / Chi-squared Non-standard (Gaussian random fields)
Generalization Overfitting with more parameters Generalization via "flat" singularities

By moving from the smooth geometry of regular models to the algebraic geometry of singularities, researchers are building a rigorous foundation for why overparameterized machines—despite their apparent complexity—are so effective at learning from data.

Papers in this version

Showing 60 of 75 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_information_geometry_of_singular_lea_11adcb51,
  title = {The Information Geometry of Singular Learning Machines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-information-geometry-of-singular-learning-machines/rdr_11adcb51be344cec9774c6ddba078641},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_information_geometry_of_singular_lea_11adcb51_v1,
  title = {The Information Geometry of Singular Learning Machines},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-information-geometry-of-singular-learning-machines/rdr_11adcb51be344cec9774c6ddba078641/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}