Reconciling modern machine learning and the bias-variance trade-off

Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik MandalarXiv 2018 · arXiv

This seminal paper by Belkin et al. addresses one of the most persistent paradoxes in modern artificial intelligence: why do massive neural networks, which theoretically should be "overfitting," actually perform better than smaller ones? By introducing the "double descent" risk curve, the authors provide a framework that reconciles classical statistical theory with the observed success of modern deep learning.

The Bias-Variance Paradox

For decades, the "Bias-Variance Trade-off" was the foundational rule of machine learning. It taught that as a model becomes more complex, its "bias" (error from being too simple) decreases, but its "variance" (sensitivity to specific noise in the training data) increases. This creates a characteristic U-shaped curve where the goal is to find the "sweet spot" before the model starts overfitting.

However, modern practice often ignores this rule. Practitioners routinely train deep neural networks with far more parameters than data points, achieving zero training error—a state known as interpolation. According to classical theory, these models should generalize poorly. Yet, in reality, they often perform better than smaller models. This paper seeks to bridge this gap: how can we have a single theory that explains both the classical "sweet spot" and the modern "bigger is better" reality?

The double descent risk curve reconciles the classical U-shaped bias-variance trade-off (a) with the performance of over-parameterized models (b).

The Double Descent Phenomenon

The authors conducted an extensive empirical study across several model classes, including Random Fourier Features (RFF), fully connected neural networks, and ensemble methods like Random Forests. Their key innovation was looking beyond the interpolation threshold—the point where the number of parameters is just enough to fit the training data perfectly.

They discovered that the risk curve does not stop at the classical U-shape. Instead, it follows a "double descent" pattern:

  1. The Classical Regime: Risk initially decreases as capacity increases, then begins to rise as the model starts to overfit noise (the standard U-shape).
  2. The Interpolation Threshold: A peak in test error occurs when the number of parameters is roughly equal to the number of data points. At this threshold, the model is forced to find a highly convoluted function to fit every data point perfectly.
  3. The Modern Regime: Beyond the threshold, as more parameters are added, the test error begins to descend again, often reaching levels lower than the classical "sweet spot."

Mechanics of the Second Descent

Why does error drop in the over-parameterized regime? The authors posit that common training algorithms (like SGD or minimum-norm solvers) have an inductive bias toward "simpler" or "smoother" functions. In the over-parameterized regime, the model has more "freedom." It can choose from an infinite number of functions that fit the data. As capacity increases, the algorithm can find even smoother functions that interpolate the data, leading to better generalization.

Empirical results for Random Fourier Features (RFF) showing how the function norm peaks at the interpolation threshold, corresponding to the spike in test risk.

As shown in Figure 6, the norm of the solution for RFF peaks exactly at the interpolation threshold. This high norm indicates a highly oscillatory function, which explains the spike in test risk. Beyond this point, the norm decreases, suggesting that the model is finding "flatter" and more stable solutions despite having more parameters.

Key Results and Implications

The study utilized standard benchmarks like MNIST, CIFAR-10, and TIMIT to demonstrate the ubiquity of this phenomenon. Key findings include:

  • Superiority of Over-parameterization: In many experiments, the test risk in the modern regime was lower than the best risk achievable in the classical regime.
  • Neural Network Behavior: For fully connected networks, the interpolation peak occurs when the number of parameters equals the number of samples times the number of classes ($n \times K$). Beyond this, test error consistently improved.
  • Implicit Regularization: The results suggest that over-parameterization acts as a form of implicit regularization, allowing models to fit data perfectly while maintaining smoothness.

Conclusion

Belkin et al. argue that a new mathematical foundation is needed for "interpolating" estimators. Rather than relying solely on the classical bias-variance trade-off, future theory must focus on the properties of the function space and the inductive biases of optimization algorithms. The double descent curve provides the first unified map for understanding why modern, massive models are not just a violation of old rules, but a new frontier of performance.

Summary updated

Abstract

Breakthroughs in machine learning are rapidly changing science and society, yet our fundamental understanding of this technology has lagged far behind. Indeed, one of the central tenets of the field, the bias-variance trade-off, appears to be at odds with the observed behavior of methods used in the modern machine learning practice. The bias-variance trade-off implies that a model should balance under-fitting and over-fitting: rich enough to express underlying structure in data, simple enough to avoid fitting spurious patterns. However, in the modern practice, very rich models such as neural networks are trained to exactly fit (i.e., interpolate) the data. Classically, such models would be considered over-fit, and yet they often obtain high accuracy on test data. This apparent contradiction has raised questions about the mathematical foundations of machine learning and their relevance to practitioners. In this paper, we reconcile the classical understanding and the modern practice within a unified performance curve. This "double descent" curve subsumes the textbook U-shaped bias-variance trade-off curve by showing how increasing model capacity beyond the point of interpolation results in improved performance. We provide evidence for the existence and ubiquity of double descent for a wide spectrum of models and datasets, and we posit a mechanism for its emergence. This connection between the performance and the structure of machine learning models delineates the limits of classical analyses, and has implications for both the theory and practice of machine learning.
BibTeX
@misc{belkin2018reconciling,
  author = {Mikhail Belkin and Daniel Hsu and Siyuan Ma and Soumik Mandal},
  title = {{Reconciling modern machine learning and the bias-variance trade-off}},
  year = {2018},
  eprint = {1812.11118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.11118},
}
Figure