Lightlike Neuromanifolds, Occam's Razor and Deep Learning

Ke Sun, Frank Nielsen

Introduction

In classical statistics, the Bayesian Information Criterion (BIC) suggests that model complexity scales with the number of parameters. By this logic, deep neural networks (DNNs) with millions of weights should be prone to extreme overfitting. However, empirical evidence shows that larger DNNs often generalize better than smaller ones. This "overparameterization paradox" arises because DNNs are singular models. Unlike regular models where every parameter change alters the prediction, DNNs contain many weight combinations that yield identical outputs. This paper provides a rigorous geometric framework to explain why these massive networks effectively obey Occam’s Razor.

The Geometry of Singular Models

The authors shift the perspective from simple parameter counting to Information Geometry. They model the parameter space as a "lightlike manifold" ($\mathcal{M}$). In physics, lightlike paths have zero length in spacetime; in this context, "lightlike" directions represent paths in the parameter space where changing weights does not change the model's output.

Geometric representation of a lightlike neuromanifold where a null curve represents equivalent models.

As shown in the figure above, a "null curve" on the manifold represents a set of equivalent models. Because the Fisher Information Matrix (FIM) is rank-deficient in these singular regions, the network has "room to move" without increasing its actual predictive complexity. The authors define a local dimensionality measure $d(\theta)$ based on the rank of the FIM, representing the true "effective" degrees of freedom.

Measuring Complexity via the Fisher Information Matrix

To derive a mathematical version of Occam’s Razor, the paper analyzes different forms of the Fisher Information Matrix. The complexity of the network is intrinsically linked to the spectral density of these matrices.

Comparison of different Fisher Information Matrix (FIM) notations and their properties.

The table above distinguishes between the true FIM, the empirical FIM (computable from data), and the observed FIM. The authors use these definitions to derive a new version of the Minimum Description Length (MDL) principle. By applying Taylor expansions and $f$-means, they bound the complexity of the network under both Gaussian and Jeffreys’ priors.

Key Insights: Singularity as a Feature

The core contribution of this work is the realization that singularities—often viewed as mathematical nuisances—are actually the secret to generalization.

  1. Negative Complexity: The terms in the new MDL formula associated with singularities actually subtract from total complexity. As a network grows wider or deeper, it creates more singular dimensions, which can paradoxically lower the "geometric complexity."
  2. The Sample Size Bound: A striking theoretical result is that the local dimensionality (effective parameters) is bounded by the number of training samples ($N$) and output classes ($m$), regardless of the raw parameter count.
  3. Justification for Flat Minima: The framework provides a formal geometric proof for the intuition that "flat minima" (regions where the FIM has many near-zero eigenvalues) lead to better generalization. These flat regions correspond to higher-dimensional singularities that simplify the model in an information-theoretic sense.

Conclusion

By modeling DNNs as lightlike neuromanifolds, the authors demonstrate that the "true" complexity of a deep network is much lower than its parameter count suggests. This geometric approach resolves the overparameterization paradox, showing that the inherent symmetries and singularities of deep learning architectures act as a natural pruning mechanism, enforcing a mathematical version of Occam's Razor.

Summary updated

BibTeX
@inproceedings{sun2025lightlike,
  author = {Ke Sun and Frank Nielsen},
  title = {{Lightlike Neuromanifolds, Occam's Razor and Deep Learning}},
  booktitle = {Information},
  year = {2025},
  doi = {10.1007/s41884-025-00167-2},
}
Figure