Statistical Inference in Singular Parameter Spaces

51 concepts29 papers2 proposals

Standard statistical theory relies on a fundamental assumption: the model is "regular." In a regular model, the mapping from parameters to probability distributions is one-to-one, and the Fisher Information Matrix (FIM) is positive definite. This allows us to treat the parameter space as a smooth Riemannian manifold, where local neighborhoods look like flat Euclidean space.

However, modern architectures like neural networks and mixture models are inherently singular. As Sumio Watanabe argues in foundational work, these models contain points where the FIM is non-invertible. At these singularities, the geometry of the parameter space breaks down into cusps, self-intersections, or sharp corners. When regularity fails, the classical tools we use to evaluate models—such as the Bayesian Information Criterion (BIC), the Akaike Information Criterion (AIC), and the Laplace approximation—become mathematically invalid.

The Geometry of Non-Regularity

In a regular model, if you perturb the parameters slightly, the resulting probability distribution changes in a predictable, linear way. In a singular model, this relationship vanishes. For example, in a neural network, if a hidden unit’s incoming weights are all zero, changing its outgoing weights has no effect on the output distribution. The parameters are "non-identifiable."

Mathias Drton demonstrates in Likelihood ratio tests and singularities that these geometric features—like self-intersections—fundamentally alter the behavior of statistical tests. Instead of the local geometry being a flat tangent space, it becomes a tangent cone. Because a cone can capture more directions of movement than a flat plane, the likelihood ratio test (LRT) statistic no longer follows a standard chi-squared distribution. Instead, it converges to a more complex distribution dictated by the shape of the cone, which can lead to incorrect p-values if the singularity is ignored.

Why Singularities Cause Divergence

The presence of singularities isn't just a technical nuisance; it changes how models learn from data. Kenji Fukumizu explains in Statistical Estimation in Singular Models that in high-capacity models like multilayer perceptrons, the likelihood ratio statistic can actually diverge logarithmically with the sample size $n$.

In a regular model, the statistic is bounded because the model's complexity is fixed. But near a singularity, the model's "effective" capacity to fit noise increases as more data is observed. This flexibility allows the model to find "lucky" fits in the messy corners of the parameter space, causing the training error to drop faster than the generalization error can keep up. As Amari et al. show, this breaks the classical symmetry between training and generalization error, rendering AIC and Minimum Description Length (MDL) unreliable.

Resolving Singularities with Algebraic Geometry

To fix this, Sumio Watanabe introduced Singular Learning Theory (SLT). The core insight, detailed in Algebraic Information Geometry for Learning Machines with Singularities, uses a tool from algebraic geometry called Hironaka’s resolution of singularities.

Essentially, this mathematical "blow-up" maps the messy, singular parameter space onto a higher-dimensional smooth manifold. This transformation allows researchers to calculate the true asymptotic behavior of the marginal likelihood. The key metric that emerges is the Real Log Canonical Threshold (RLCT). As Wei et al. argue, the RLCT serves as the true measure of a model's effective complexity. It explains why "flat" minima in deep learning generalize so well: they correspond to regions with specific singular structures where the RLCT is small, meaning the model is effectively simpler and less prone to overfitting than its total parameter count suggests.

Practical Consequences for Model Selection

The failure of standard criteria has led to the development of new tools for non-regular models:

  • Mixture Models: When you use too many components to fit simple data, the model becomes overspecified. Khamaru et al. proved that this causes the Expectation-Maximization (EM) algorithm to slow down exponentially. To handle this, Fujimaki and Morinaga proposed Factorized Asymptotic Bayesian (FAB) inference, which uses SLT to automatically prune redundant components that BIC would fail to detect.
  • Causal Discovery: Causal models with unobserved variables often have sharp boundaries and singularities. Bellot et al. leverage the Widely Applicable Bayesian Information Criterion (WBIC)—a singular-aware score—to distinguish between causal structures that look identical under regular statistical tests.
  • Universal Testing: To avoid the fragility of regularity assumptions altogether, Ramdas et al. developed the Universal Likelihood Ratio Test (ULRT). By using split-sample techniques, they provide a test that remains valid even in "messy" non-convex hypothesis spaces where traditional LRTs break down.

By moving beyond the assumption of smoothness, this research direction provides a rigorous way to understand how complex, overparameterized machines actually extract information from data.

Key Foundational Work

Algebraic Information Geometry for Learning Machines with Singularities (2000) — contributed 1 concept elements to this direction.
Algebraic Analysis for Non-regular Learning Machines (1999) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_parameter_space_singularities_e71b1714,
  title = {parameter space singularities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-space-singularities/rdr_e71b1714f94746df80421db108f5a06b},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_parameter_space_singularities_e71b1714_v1,
  title = {parameter space singularities},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/parameter-space-singularities/rdr_e71b1714f94746df80421db108f5a06b/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake