Modern deep neural networks (DNNs) often possess millions of parameters, far exceeding the number of training samples. According to classical learning theory, such as VC-dimension, these models should suffer from severe overfitting. However, in practice, DNNs generalize remarkably well. This paper investigates the existence and properties of "points of extreme overfitting"—weight configurations that yield near-perfect training accuracy but near-zero test accuracy. By explicitly finding these points, the authors provide empirical evidence for why standard optimization rarely encounters them.
Methodology: Constructing Extreme Overfitting
To identify these catastrophic minima, the authors employ a novel data manipulation strategy rather than modifying the optimization algorithm itself. The process involves three primary steps:
- Test Set Corruption: The labels of the standard test set are intentionally altered to be incorrect.
- Augmentation and Concatenation: This corrupted test set is duplicated until its size matches the original training set.
- Combined Training: The original training set and the duplicated, corrupted test set are concatenated into a single "corrupted training set."
By training standard architectures (MLP, CNN, ResNet) on this combined dataset, the network is forced to memorize both the correct training labels and the intentionally incorrect test labels. Because modern networks are highly overparameterized, they have the capacity to memorize this entire combined dataset perfectly.
Empirical Results
The experiments were conducted on MNIST, Fashion MNIST, and CIFAR-10. In all cases, the models successfully reached points of extreme overfitting. As shown in the training curves, the training loss decreases steadily while the test loss increases dramatically, reflecting the model's perfect memorization of the corrupted labels.
Interestingly, advanced architectures like ResNet reached these extreme overfitting points faster than simpler models. This suggests that the increased capacity and architectural innovations that aid generalization also facilitate faster memorization when the objective is corrupted. Even with $L_2$ regularization (weight decay), the optimizers were still able to find these points of extreme memorization.
The Distance Hypothesis and Landscape Geometry
A key finding of this study is the spatial relationship between these "bad" minima and standard solutions. The authors measured the Euclidean norm of the network weights and found that extreme overfitting points are located significantly further from the initialization point than the weights found during normal training. This suggests that standard optimizers like SGD or Adam are "lazy"—they tend to settle into the nearest viable minima, which happen to be those that generalize well.
Furthermore, these catastrophic minima appear to be extremely sharp and narrow. When a model at an extreme overfitting point is retrained on uncorrupted data, the optimizer quickly escapes the bad minimum and converges to a standard, well-generalizing solution. This implies that while these "sad points" exist, they occupy a very small volume of the loss landscape compared to broader, generalizing regions.
Conclusion
This empirical study confirms that extreme overfitting points are a real feature of the neural network loss landscape. However, their distance from standard initialization and their narrow geometry explain why they are rarely encountered in practice. These findings reinforce the idea that the implicit bias of standard optimization algorithms toward "closer" and "flatter" minima is a primary driver of the remarkable generalization observed in deep learning.
Summary updated