In classical statistics, we often assume that models are "regular." This means that every set of parameters corresponds to a unique probability distribution and that the loss landscape looks like a tidy, quadratic bowl near the optimum. In this world, the Fisher Information Matrix (FIM)—which measures the curvature of the log-likelihood—is positive definite and invertible.
However, modern architectures like neural networks and mixture models are fundamentally singular. As Sumio Watanabe rigorously established in Algebraic Information Geometry for Learning Machines with Singularities, these models contain regions where the FIM becomes degenerate. This isn't just a mathematical technicality; it is the reason why traditional tools like the Bayesian Information Criterion (BIC) fail to predict how deep networks learn and generalize.
The Geometry of Redundancy
Why does the Fisher Information Matrix become singular? The culprit is usually redundancy. In a multilayer perceptron, if two hidden units have the same weights, or if a weight is set to zero, the model's output remains the same even if you change those parameters in certain directions.
Kenji Fukumizu demonstrated in Active Learning in Multilayer Perceptrons that these "mathematical collapses" prevent algorithms from calculating optimal data sampling because they rely on inverting the FIM. When the matrix is singular, the "curvature" in those redundant directions is zero—the landscape is perfectly flat.
Shun-ichi Amari and colleagues further characterized these regions as geometric cones in Geometrical Singularities in the Neuromanifold of Multilayer Perceptrons. Because of symmetries (like swapping two neurons), the parameter space isn't a smooth manifold but a collection of intersecting sheets. At these intersections, the standard "quadratic bowl" approximation of the loss function vanishes.
Why Singularities Cause Optimization Plateaus
If you have ever watched a neural network's training loss stall for thousands of iterations before suddenly dropping, you have likely encountered a singularity.
In Singular Learning of Deep Multilayer Perceptrons for EEG-Based Emotion Recognition, Weili Guo and others show that when the FIM becomes singular, gradient-based optimization loses its directional guidance. The algorithm "wanders" in these flat regions because the gradient provides no information on which way to go.
This creates a paradox for second-order methods like Natural Gradient Descent (NGD). NGD tries to navigate the landscape by inverting the FIM to account for its curvature. But as Wu Lin and Frank Nielsen note in Tractable structured natural-gradient descent using local parameterizations, this inversion becomes numerically unstable or impossible at singular points.
The Real Log Canonical Threshold
If parameter count isn't a reliable measure of complexity for singular models, what is? Susan Wei and Hui Li argue in Deep Learning Is Singular, and That's Good that we must use Singular Learning Theory (SLT).
The key metric in SLT is the Real Log Canonical Threshold (RLCT). While classical theory says a model's complexity is roughly half the number of its parameters ($d/2$), the RLCT provides a "singular-aware" complexity measure. In overparameterized models, the RLCT is often much smaller than the parameter count.
This explains the "Occam's Razor" of deep learning: massive models generalize well because their effective complexity (the RLCT) is low. As Edmund Lau and Daniel Murfet describe in The Local Learning Coefficient: A Singularity-Aware Complexity Measure, the RLCT measures how "pointy" or "flat" a singularity is. "Flatter" singularities have smaller RLCTs, meaning they are simpler and more likely to generalize.
Generalization and Double Descent
The singular nature of these models also provides a geometric explanation for Double Descent, where increasing model size beyond the point of perfect training fit actually improves test performance.
Mahito Sugiyama and Prasad Cheema show in A Geometric Look at Double Descent Risk that as dimensionality increases, the "model volume" can actually shrink or saturate due to these singularities. Similarly, Ke Sun and Frank Nielsen propose in A geometric modeling of Occam’s razor in deep learning that singularities provide a "negative complexity" that counteracts the explosion of parameters.
Summary of the Shift
| Feature | Regular Statistical Theory | Singular Learning Theory |
|---|---|---|
| Fisher Information | Positive definite, invertible | Singular, non-invertible |
| Loss Landscape | Quadratic bowl | Cones, flat valleys, intersections |
| Complexity Metric | Parameter count ($d/2$) | Real Log Canonical Threshold (RLCT) |
| Asymptotics | Gaussian / Chi-squared | Non-standard (Gaussian random fields) |
| Generalization | Overfitting with more parameters | Generalization via "flat" singularities |
By moving from the smooth geometry of regular models to the algebraic geometry of singularities, researchers are building a rigorous foundation for why overparameterized machines—despite their apparent complexity—are so effective at learning from data.