In classical statistics, we are taught to fear overfitting. The "U-shaped" risk curve suggests that as a model becomes complex enough to fit the noise in a dataset, its generalization performance must suffer. Yet, modern deep learning models routinely interpolate—perfectly fitting every noisy training point—while still generalizing excellently.
Recent research has reconciled these views by decomposing the "excess risk" of a model into two distinct forces: approximation error (bias) and estimation error (variance). By analyzing how these components evolve in high-dimensional spaces, researchers are uncovering the specific conditions that make overfitting "benign" rather than "malignant."
The Anatomy of the Risk Curve
To understand why overparameterized models work, we must look at what happens to bias and variance as we increase model capacity. In On the Double Descent of Random Features Models Trained with SGD, Fanghui Liu and Johan A. K. Suykens demonstrate that the "double descent" phenomenon is driven by a specific interaction: while the bias (the model's ability to represent the true signal) typically decreases monotonically as capacity grows, the variance (sensitivity to training noise) exhibits a sharp peak at the "interpolation threshold"—the point where the number of parameters exactly matches the number of samples.
Beyond this threshold, the variance begins to decrease again. This happens because the model has "room" to distribute the noise across many redundant dimensions. As Junhyung Park and Patrick Blöbaum argue in A Classical View on Benign Overfitting: The Role of Sample Size, the classical U-shaped curve and modern benign overfitting are two sides of the same coin, governed by how model complexity and sample size interact to manage this estimation error.
Why Noise Fitting Can Be Benign
The core mystery is how a model can fit $y = f(x) + \epsilon$ (where $\epsilon$ is random noise) without the noise corrupting the prediction of $f(x)$. The intuition is that in very high dimensions, the model can satisfy the requirement to hit every noisy data point by using its "extra" parameters to create high-frequency, low-energy spikes that vanish almost everywhere except at the training points.
Junhyung Park and Shiva Prasad Kasiviswanathan provided the first theoretical proof in Benign Overfitting for Regression with Trained Two-Layer ReLU Networks that finite-width ReLU networks can achieve this. They show that the network can perfectly fit noisy data while the "estimation error" remains small, effectively "hiding" the noise in the model's weights without degrading its approximation of the underlying function.
The Inductive Bias Paradox
A surprising finding in this field is that "stronger" inductive biases—which we usually think of as helpful—can actually make interpolation harmful.
In Fast rates for noisy interpolation require rethinking the effect of inductive bias, Konstantin Donhauser and Fanny Yang show that if a model has a very strong sparsity bias (like an $\ell_1$ norm), it is forced to fit noise by assigning massive weights to just a few features. This "concentrates" the noise, causing the variance to explode. Conversely, a "moderate" bias ($\ell_p$ norms where $1 < p < 2$) allows the model to spread the noise more thinly across many features, leading to faster generalization.
This theme is echoed by Michael Aerni and colleagues in Strong inductive biases provably prevent harmless interpolation. They demonstrate that structural constraints, such as highly localized convolutional filters, prevent "harmless" interpolation. Because the model is so constrained, it cannot "hide" the noise in redundant dimensions; instead, it must distort its representation of the signal to accommodate the noise, leading to a "malignant" increase in variance.
Architectural Levers for Variance Control
If variance is the primary enemy in the interpolation regime, how can we design architectures to suppress it?
- Skip Connections: In Impact of Bottleneck Layers and Skip Connections on the Generalization of Linear Denoising Autoencoders, Jeroen van der Ham and Debarghya Ghoshdastidar show that skip connections act as a variance-reduction tool. While bottleneck layers introduce a classical bias-variance trade-off, skip connections smooth out the risk curve, making models more robust to noise even when they are heavily overparameterized.
- Data Splitting: Nicole Mücke and Enrico Reiss suggest in Data-splitting improves statistical performance in overparameterized regimes that splitting data across multiple models can actually improve performance. By averaging the outputs of models trained on different subsets, we can reduce the high estimator variance that naturally arises when individual models try to interpolate noise.
Generalization Under Shift
Finally, the "benign-ness" of overfitting is not just a property of the model, but also of the data distribution. Neil Mallinar and Bin Yu explore this in Minimum-Norm Interpolation Under Covariate Shift. They find that the degree of overparameterization determines whether a shift in the test distribution is beneficial or malignant. In high-dimensional settings, the model's sensitivity to specific directions in the data determines whether the "noise spikes" it created during training will interfere with new, unseen data.