Generalization through Minimum Norm Interpolation

This direction is merged into another direction: overparameterization generalization paradox. Its last version stays readable and citable here.

Version 1 of 163 papers65 conceptsfrozen 2026-10-04

Classical statistical wisdom suggests a strict trade-off between bias and variance: as a model becomes more complex, it eventually begins to "overfit," fitting the noise in the training data at the expense of its ability to generalize to new examples. However, modern deep learning models frequently operate in an "over-parameterized" regime—where the number of parameters far exceeds the number of data points—and yet they generalize remarkably well even when they achieve zero training error.

This phenomenon is driven by minimum-norm interpolation, a mechanism where optimization algorithms naturally select the "simplest" possible function among the infinite set of functions that perfectly fit the data.

The Implicit Bias of Optimization

In an underdetermined system, where there are more variables than observations, there are infinitely many ways to fit the training data perfectly. Traditional regularization (like Ridge regression) explicitly penalizes large weights to prevent overfitting. As Cynthia Rudin established in Stability Analysis for Regularized Least Squares Regression, these regularized models converge to the simplest function in the hypothesis space as noise vanishes.

In modern over-parameterized settings, we often skip explicit regularization. Instead, algorithms like gradient descent exhibit an implicit bias. As noted by Weinan E, Chao Ma, and Lei Wu in On the Generalization Properties of Minimum-norm Solutions for Over-parameterized Neural Network Models, these algorithms tend to converge to the solution with the minimum functional norm. By selecting the solution with the smallest weights (the $L_2$ norm), the model maintains a degree of smoothness that prevents the "memorization" of noise from ruining predictions on unseen data.

Benign Overfitting and the Role of High Dimensions

The ability to fit noisy data perfectly without sacrificing generalization is known as benign overfitting. This occurs when the noise in the training data is effectively "absorbed" by the many extra dimensions available in an over-parameterized model.

Li Zhu and Weijie J Su demonstrate in Benign Overfitting and Noisy Features that noise acts as an implicit regularizer. In high-dimensional spaces, the minimum-norm estimator distributes the error across many "tail" dimensions (features with low variance), leaving the primary signal intact in the dominant dimensions.

However, this "magic" is not universal. Alexander Rakhlin and Xiyu Zhai proved in Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon that for certain kernels, interpolation is fundamentally inconsistent in low dimensions. The model must have enough "room" in high-dimensional space to bury the noise without distorting the underlying signal.

The Double Descent Risk Curve

The transition from classical regimes to over-parameterized regimes is best visualized through the Double Descent curve. As model complexity increases:

  1. Classical Regime: Risk follows a U-shape, decreasing as the model learns the signal and increasing as it begins to overfit.
  2. Interpolation Threshold: Risk peaks exactly when the number of parameters equals the number of data points. At this "critical" point, the model is forced to use every bit of its capacity to fit the noise, leading to highly unstable solutions.
  3. Over-parameterized Regime: As parameters continue to increase, the risk drops again.

Saharon Rosset, Ryan J. Tibshirani, and Trevor Hastie show in Surprises in High-Dimensional Ridgeless Least Squares Interpolation that in this second descent, the minimum-norm solution becomes increasingly stable. By spreading the weights across more features, the model becomes less sensitive to any single noisy observation.

Beyond Linear Regression

While much of the theory is grounded in linear models, the principles of minimum-norm interpolation extend to complex architectures:

By shifting the focus from "how many parameters a model has" to "how the model uses its capacity," researchers have reconciled the success of massive neural networks with the fundamental principles of statistical learning. The minimum-norm solution is the bridge that allows models to memorize everything while still understanding the big picture.

Papers in this version

Showing 60 of 63 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalization_through_minimum_norm_inte_f23c6678,
  title = {Generalization through Minimum Norm Interpolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-through-minimum-norm-interpolation/rdr_f23c6678b04f4519a1e72ad399254a53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalization_through_minimum_norm_inte_f23c6678_v1,
  title = {Generalization through Minimum Norm Interpolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalization-through-minimum-norm-interpolation/rdr_f23c6678b04f4519a1e72ad399254a53/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}