High Dimensionality as a Requirement for Benign Overfitting

This direction is merged into another direction: overparameterization generalization paradox. Its last version stays readable and citable here.

Version 1 of 119 papers39 conceptsfrozen 2026-10-04

In classical statistics, the "bias-variance tradeoff" suggests that a model perfectly fitting noisy training data—a process called interpolation—should generalize poorly. Yet, modern deep learning models often interpolate their training sets and still perform exceptionally well. This phenomenon, termed benign overfitting, has sparked a massive effort to rewrite the rules of statistical learning.

However, recent research reveals a crucial caveat: the "magic" of benign overfitting is not a universal property of all models. Instead, it is a high-dimensional phenomenon. In fixed, low-dimensional spaces, the classical wisdom holds true: interpolating noisy data with popular kernel methods is mathematically guaranteed to fail.

The Geometry of a Spike

To understand why dimensionality matters, we must look at how a model behaves when it is forced to hit every noisy data point exactly.

Imagine you are using a smooth kernel, like a Gaussian or Laplace kernel, to fit a set of points. If the data is noisy, the "true" function might be a simple line, but the training labels are scattered above and below it. To interpolate these points, a minimum-norm solution must deviate from the smooth true function to "reach" each noisy label.

In low dimensions, Daniel Beaglehole, Mikhail Belkin, and Parthe Pandit show that these deviations take the form of "spikes." Because the kernel prefers smoothness, it stays as close to the true function as possible for most of the space, only creating sharp, local fluctuations to satisfy the interpolation constraint at the data points.

As you add more data, these spikes do not smooth out. Instead, they become more frequent. In On the Inconsistency of Kernel Ridgeless Regression in Fixed Dimensions, the authors demonstrate that these spikes prevent the estimator from ever converging to the ground truth. The model becomes a "spiky" mess that fits the noise perfectly but fails to capture the underlying signal.

The Inconsistency of Low-Dimensional Interpolation

This failure is formally known as inconsistency: the estimation error remains bounded above zero even as the number of training samples approaches infinity.

Alexander Rakhlin and Xiyu Zhai proved this for the Laplace kernel in Consistency of Interpolation with Laplace Kernels is a High-Dimensional Phenomenon. They found that in fixed dimensions, there is a fundamental tradeoff:

  1. If the kernel bandwidth is small, the function is too localized to generalize.
  2. If the bandwidth is large, the "energy" (RKHS norm) required to fit the noise grows too slowly to allow the spikes to vanish.

This result was extended to Gaussian kernels by Marko Medvedev, Gal Vardi, and Nati Srebro in Overfitting Behaviour of Gaussian Kernel Ridgeless Regression. They found that in fixed dimensions, ridgeless (interpolating) Gaussian regression often performs worse than a "null predictor" that simply guesses the average label. No amount of bandwidth tuning can save the model from this catastrophic overfitting unless the dimensionality of the input also increases.

Why High Dimensions Save the Day

If interpolation fails so spectacularly in low dimensions, why does it work in modern machine learning? The answer lies in the "blessing of dimensionality."

In high-dimensional spaces, the geometric relationship between points changes. As Shao-Bo Lin and Xingping Sun argue in Kernel Interpolation of High Dimensional Scattered Data, kernel matrices become better conditioned as dimensionality increases. In high dimensions, points are naturally "spread out," allowing the model to fit noise without creating the same kind of destructive local spikes that ruin generalization in 2D or 3D.

Daniel Hsu and Mikhail Belkin provided some of the first theoretical bounds in Overfitting or perfect fitting?, showing that interpolating rules can be consistent if the feature space is sufficiently high-dimensional. In these regimes, the "noise" is effectively distributed across many dimensions, allowing the "signal" to remain visible.

Harmful Overfitting in Sobolev Spaces

The failure of interpolation in low dimensions isn't just a quirk of Gaussian or Laplace kernels; it appears to be a fundamental property of smoothness-based regularization.

In Harmful Overfitting in Sobolev Spaces, Kedar Karhadkar and Guido Montufar examine models that minimize Sobolev norms (which penalize derivatives to ensure smoothness). They prove that in fixed dimensions, picking the "smoothest" function that interpolates noisy data is a recipe for harmful overfitting. The generalization error remains high even with infinite data because the requirement to be smooth and hit noisy points simultaneously is too restrictive in low-dimensional space.

Implications for Neural Networks

This research has profound implications for how we understand neural networks. Weinan E and Lei Wu have shown that highly over-parameterized neural networks often behave like fixed-kernel methods (the Neural Tangent Kernel regime).

If these networks are essentially performing kernel interpolation, they are subject to the same dimensionality constraints. In A Comparative Analysis of the Optimization and Generalization Property of Two-layer Neural Network and Random Feature Models, they argue that massive over-parameterization can lead to a "curse of dimensionality" where the model generalizes poorly unless the target function is very simple or the input dimension is very high.

To bypass this, we may need more than just "benign overfitting." For example, Dan Qiao and Daniel Soudry suggest in Stable Minima Cannot Overfit in Univariate ReLU Networks that specific optimization choices, like using large learning rates, can act as an implicit regularizer that picks out smoother solutions than pure interpolation would suggest, potentially avoiding the "spiky" failures of kernel methods in low dimensions.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_high_dimensionality_as_a_requirement_for_3d29f596,
  title = {High Dimensionality as a Requirement for Benign Overfitting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensionality-as-a-requirement-for-benign-overfitting/rdr_3d29f59655c74d3993d8c8e4ae255d2a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_high_dimensionality_as_a_requirement_for_3d29f596_v1,
  title = {High Dimensionality as a Requirement for Benign Overfitting},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/high-dimensionality-as-a-requirement-for-benign-overfitting/rdr_3d29f59655c74d3993d8c8e4ae255d2a/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}