The Effective Dimensionality of Neural Networks

43 concepts28 papers1 proposal

In classical statistics, a model’s complexity is usually equated with its number of parameters. A model with more weights is assumed to be more flexible, more prone to overfitting, and more demanding of data. However, modern deep learning has shattered this intuition: massive neural networks with millions of parameters often generalize better than smaller ones, a phenomenon known as "double descent."

The resolution to this paradox lies in the concept of effective dimensionality. Rather than counting every weight in a network, researchers argue that we should measure how many "degrees of freedom" are actually constrained by the training data.

The Geometry of Constraint

The intuition behind effective dimensionality is that not all parameters are equally important. In a typical loss landscape, many directions in parameter space are "flat," meaning that moving the weights in those directions has almost no effect on the model’s predictions. Conversely, "sharp" directions represent parameters that are tightly constrained by the data.

MacKay and Moody were among the first to formalize this using the Hessian matrix of the loss function. The Hessian captures the curvature of the landscape; its eigenvalues tell us how sensitive the loss is to changes in different directions. MacKay proposed that the effective number of parameters is determined by the number of large eigenvalues in the Hessian—essentially, the number of directions where the data provides enough information to "pin down" the weights.

Wilson et al. revived this idea to explain why overparameterized models generalize. They show that as models grow larger, their effective dimensionality often stays low or even decreases, because the optimization process finds solutions in increasingly flat regions of the loss landscape.

Measuring Complexity through Curvature

Several mathematical formulations have been proposed to capture this "soft count" of parameters:

  • Trace-based metrics: Murata et al. and Moody quantified effective complexity using the trace of the product of the gradient variance and the inverse Hessian. This metric accounts for the interaction between the network architecture and the specific noise in the data.
  • Information Matrices: In quantum machine learning, Khanal et al. use the eigenvalues of the information matrix to identify redundant or "inert" parameter directions, providing a measure of capacity that ignores noise-driven fluctuations.
  • Smoother Matrices: For models that behave like kernels, Burgess suggested using the trace of the "smoother matrix," which measures how much each training point influences the final prediction.

These measures share a common goal: they penalize a model not for having many weights, but for having many influential weights. As Kwok et al. argue, this allows for a principled way to adjust regularization dynamically, ensuring the model's complexity matches the information content of the data.

Singularities and Latent Variables

The challenge of measuring complexity becomes even more acute in "singular" models. Modern neural networks are not smooth manifolds; they contain singularities where the mapping from parameters to functions is not one-to-one. Watanabe demonstrated that in these cases, traditional metrics like the Bayesian Information Criterion (BIC) fail because the effective dimensionality is lower than the parameter count due to the geometry of these singularities.

This problem extends to causal discovery and latent variable models. When dealing with hidden variables, the relationship between the number of edges in a graph and its statistical degrees of freedom is often undefined. Améndola et al. and Ng et al. highlight that without a precise measure of effective dimension, it is difficult to balance model fit with parsimony in complex hierarchical structures.

Complexity as a Trajectory

Finally, some researchers argue that complexity is not just a property of the model architecture, but of the optimization process itself. Simsekli et al. proposed using the Hausdorff dimension of the path taken by Stochastic Gradient Descent (SGD). They found that the "fractal roughness" of the optimization trajectory—driven by heavy-tailed gradient noise—serves as an intrinsic measure of complexity. A sparser, less space-filling path correlates with better generalization, suggesting that the optimizer itself acts as a form of implicit regularization by restricting the effective dimensionality of the search space.

By shifting our focus from raw parameter counts to the geometric and algebraic properties of the loss landscape, we gain a clearer understanding of why deep learning works: it isn't about how many weights you have, but how many of them the data actually forces you to use.

Key Foundational Work

Learning Curves, Model Selection and Complexity of Neural Networks (1992) — contributed 2 concept elements to this direction.
Bayesian Model Comparison and Backprop Nets (1991) — contributed 1 concept elements to this direction.
The Effective Number of Parameters: An Analysis of Generalization and Regularization in Nonlinear Learning Systems (1991) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_effective_model_complexity_160a9f86,
  title = {effective model complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effective-model-complexity/rdr_160a9f867810446082af3b2c46dbe28f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_effective_model_complexity_160a9f86_v1,
  title = {effective model complexity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/effective-model-complexity/rdr_160a9f867810446082af3b2c46dbe28f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake