In classical learning theory, a model that perfectly fits noisy training data—reaching zero training error—is expected to generalize poorly. This "overfitting" is typically explained through uniform convergence: if a hypothesis class is complex enough to interpolate noise, the worst-case gap between training and test error should be large.
However, modern deep learning models frequently interpolate noisy data and still generalize well, a phenomenon known as "benign overfitting." This led some researchers to suggest that uniform convergence is fundamentally unable to explain modern machine learning. Recent work by Lijia Zhou, Nathan Srebro, and others suggests a different perspective: uniform convergence isn't broken; we were just applying it to the wrong sets.
The Failure of Global Bounds
Standard generalization bounds look at the entire hypothesis class $\mathcal{H}$ (for example, all linear predictors with a certain norm). They calculate the "worst-case" difference between training and test error across this whole space.
In overparameterized regimes, this global approach fails. Because the model is flexible enough to fit any labels, the hypothesis class contains many "bad" functions that fit the training data perfectly but are total noise elsewhere. As long as these bad functions exist in $\mathcal{H}$, the global uniform convergence bound remains "vacuous" (larger than the maximum possible error), telling us nothing about why our specific model generalizes.
Shrinking the Scope to the Interpolation Manifold
The core insight of this research direction is to restrict the analysis. Instead of looking at the entire hypothesis class, we should only look at the subset of functions that actually achieve zero training error on our specific dataset. This is known as interpolation-restricted uniform convergence.
In On Uniform Convergence and Low-Norm Interpolation Learning, Lijia Zhou, Danica J. Sutherland, and Nathan Srebro demonstrated that while the "global" bound might be vacuous, the bound restricted to the manifold of interpolators can be quite small. By focusing only on the predictors that the algorithm could actually choose (those with zero training error), the theory begins to align with the observed reality of benign overfitting.
Leveraging Gaussian Width and Optimistic Rates
To make these restricted bounds precise, researchers have turned to high-dimensional geometry. In Uniform Convergence of Interpolators: Gaussian Width, Norm Bounds and Benign Overfitting, Frederic Koehler, Lijia Zhou, and colleagues used the concept of Gaussian width to characterize the complexity of the interpolator set. They showed that for Gaussian linear regression, this restricted uniform convergence is sufficient to recover known consistency results for both $\ell_2$-norm (ridge-style) and $\ell_1$-norm (basis pursuit) interpolators.
Building on this, Optimistic Rates: A Unifying Theory for Interpolation Learning and Regularization in Linear Regression introduces a framework that bridges the gap between classical regularization and modern interpolation. By using "optimistic rates"—refined bounds that eliminate loose logarithmic factors—the authors show that a localized version of uniform convergence can precisely characterize risk across the entire spectrum, from models that are heavily regularized to those that perfectly fit noise.
The "Exact Gap" and Remaining Challenges
While restricting uniform convergence to interpolators prevents bounds from becoming vacuous, it does not necessarily make them perfectly tight. Zitong Yang, Song Mei, and Yu Bai explored this in Exact Gap between Generalization Error and Uniform Convergence in Random Feature Models.
They found that even when restricted to interpolators, uniform convergence bounds can still significantly overestimate the actual test error. This is because uniform convergence is still a "worst-case" measure over the interpolator set, whereas an algorithm like SGD might pick a specific "best-case" interpolator.
Furthermore, the applicability of these bounds to more complex settings remains an open question. As noted by Shira Vansover-Hager, Tomer Koren, and Roi Livni in Rapid Overfitting of Multi-Pass SGD in Stochastic Convex Optimization, many proofs of "catastrophic overfitting" in multi-pass SGD rely on constructions where uniform convergence specifically fails to hold, suggesting that the relationship between optimization paths and these restricted sets is the next frontier for the theory.