Generalization Theory for Overparameterized Models

This direction is no longer present in the current build. Its last version stays readable and citable here.

Version 1 of 170 papers70 conceptsfrozen 2026-10-04

Classical statistical wisdom suggests a trade-off: as a model's complexity increases, its training error drops, but its test error eventually rises—the familiar "U-shaped" curve of the bias-variance trade-off. However, modern deep learning has shattered this intuition. Massive neural networks with millions of parameters often achieve zero training error (interpolation) and yet generalize better than smaller models.

This research direction seeks to reconcile theory with practice by characterizing the phenomena of double descent, benign overfitting, and implicit regularization.

The Interpolation Paradox and Double Descent

The most striking departure from classical theory is the "double descent" phenomenon. As model capacity increases, the test error first follows the classical U-curve. But as the model becomes large enough to perfectly fit the training data (the interpolation threshold), the error peaks and then begins a second descent.

Mei Song provided precise asymptotic formulas for this curve in random features regression, proving that extreme overparameterization can actually be optimal. While it was long believed that this peak only occurs when the number of parameters matches the number of data points, Xinyue Li and Rishi Sonthalia have shown that these peaks can shift into the under-parameterized regime depending on the spectral properties of the data.

Why Overfitting Can Be Benign

If a model perfectly fits noisy training data, why doesn't the noise destroy its predictive power? The answer lies in benign overfitting. In high-dimensional settings, the model can "absorb" the noise into many unimportant parameter directions while still capturing the global signal.

Difan Zou, Quanquan Gu, and Sham M. Kakade characterized this for Stochastic Gradient Descent (SGD), revealing that the data's covariance structure determines whether overfitting remains "benign." Similarly, Frederic Koehler and Nathan Srebro demonstrated that uniform convergence—when restricted to the set of interpolators—can explain why these models generalize despite their complexity.

Implicit Regularization as a Hidden Hand

In overparameterized regimes, there are infinitely many solutions that achieve zero training error. Why does the optimization process pick a "good" one? Research suggests that the optimization algorithm itself—usually SGD—acts as an implicit regularizer.

Michal Derezinski and Michael W. Mahoney proved that finding the minimum-norm solution in ridgeless regression is mathematically equivalent to classical ridge regularization. Furthermore, Ohad Shamir established that gradient methods on separable data never overfit in finite time, as they are biased toward finding the maximum-margin solution. This "hidden" regularization helps models find "flat minima," a property Shixing Yu and Tong Lin have leveraged to create explicit regularizers based on the Hessian trace.

Bridging the Linear to Non-linear Gap

Much of the rigorous theory in this field—such as the work on Sketched Ridgeless Linear Regression by Xin Chen—is built on linear models. A central challenge is extending these insights to deep, non-linear architectures.

Chaoyue Liu and Mikhail Belkin have explored the "linearity" of large models through the lens of the Neural Tangent Kernel, explaining why wide networks often behave like linear approximations. Meanwhile, Wei Lin and Huiyuan Wang have begun bridging this gap for two-layer ReLU networks, uncovering a "ridge-lasso duality" that reproduces double descent in non-linear settings.

Practical Implications of Overparameterization

This theoretical shift has practical consequences for how we train and deploy models:

  • Data Splitting and Bagging: Nicole Mücke and Enrico Reiss found that data-splitting in overparameterized regimes can actually improve performance, while Jin-Hong Du showed that bagging can "monotonize" the risk curve, removing the double descent peak.
  • Privacy Risks: There is a trade-off between performance and security. Jasper Tan and Hamid Javadi proved that as models become more overparameterized to improve generalization, they also become more vulnerable to membership inference attacks.
  • Label Noise: While bigger is often better, Yihao Xue and Baharan Mirzasoleiman discovered a "final ascent" where test loss increases for extremely large models in the presence of significant label noise, suggesting that sparsity still has a role to play in messy, real-world data.

Papers in this version

Showing 60 of 70 papers.

History & citationVersion 1 · 2026-10-04

No longer present in the current build · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalization_theory_for_overparameteri_0ecb8402,
  title = {Generalization Theory for Overparameterized Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-theory-for-overparameterized-models/rdr_0ecb840203674308ad19998d39bb872f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalization_theory_for_overparameteri_0ecb8402_v1,
  title = {Generalization Theory for Overparameterized Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-theory-for-overparameterized-models/rdr_0ecb840203674308ad19998d39bb872f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}