TL;DR
This paper uncovers the "final ascent" phenomenon: in the presence of label noise, the generalization performance of overparameterized neural networks eventually degrades as model width or density increases. The authors demonstrate that while scaling up usually helps, high noise-to-sample-size ratios make intermediate-sized models optimal, and surprisingly, standard robust training methods can actually make this performance drop at large scales even worse.
What Problem Does This Paper Address?
The deep learning community has recently embraced the "double descent" phenomenon, which suggests that increasing model size beyond the point of interpolation continues to improve generalization, contradicting classical bias-variance trade-off intuition. However, most of these studies assume clean or low-noise labels.
In the real world, large-scale datasets often rely on noisy sources like web-crawling or crowd-sourcing. This paper investigates a critical gap: How does label noise interact with model scaling (width and density)? The authors challenge the "bigger is always better" mantra by showing that under sufficient noise, the test loss curve doesn't just descend—it starts climbing again.
Methods
The authors employ a two-pronged approach:
- Theoretical Analysis: They analyze random feature ridge regression in a high-dimensional asymptotic limit. By decomposing the expected risk (test loss) into Bias², Variance$_{clean}$, and Variance$_{noise}$, they mathematically characterize how label noise alters the shape of the variance curve.
- Model Density as a New Dimension: Beyond just model width, they introduce model density—the fraction of trainable weights—as a variable. They provide the first theoretical characterization of how reducing density (randomly dropping parameters) affects generalization under noise.
- Empirical Validation: They conduct extensive experiments across various architectures (MLPs, ResNets, Vision Transformers, and InceptionResNet-v2) to see if their theoretical findings hold in practical deep learning scenarios.
Datasets
The study uses a mix of synthetic and real-world noise:
- MNIST: Handwritten digits with synthetic symmetric noise.
- CIFAR-10 / CIFAR-100: Object recognition with both symmetric and asymmetric (class-dependent) synthetic noise.
- Stanford Cars: A fine-grained classification dataset using "Web Noise"—real-world label errors introduced by automated web searches.
Key Insights and Ideas
-
The Final Ascent: The most striking finding is that the test loss curve can exhibit a U-shape or a "decreasing-increasing-decreasing-increasing" pattern. At very high widths, the noise-induced variance dominates, causing generalization to worsen.
Figure 1: Risk decomposition showing the final ascent in test loss as width increases. -
Variance Shape Transition: The authors show that label noise fundamentally changes the behavior of model variance. While clean variance is typically unimodal, noise-induced variance monotonically increases with width. At high noise levels, the total variance develops an "increasing-decreasing-increasing" shape.
Figure 2: Decomposition of variance showing how noise-induced variance (dotted line) drives the final ascent. -
The Regularization Paradox: Intuitively, one might expect stronger regularization ($\ell_2$ or weight decay) to mitigate noise. However, the authors find that stronger regularization actually exacerbates the final ascent, making the performance gap between intermediate and large models even wider.
Figure 3: Stronger regularization (λ₀) makes the final ascent more pronounced. -
Wider but Sparser is Better: The paper suggests that for a fixed parameter budget, a wider model with lower density (more sparsity) generalizes better under label noise than a narrower, dense model.
Key Results
- Noise Thresholds: The final ascent can appear with as little as 20% label noise on datasets like CIFAR-10.
- Density vs. Regularization: Adjusting model density can achieve lower test loss than simply tuning $\ell_2$ regularization. In experiments on Stanford Cars, reducing density significantly improved performance under 30% and 70% noise.
- Robust Algorithms: State-of-the-art robust training methods (like ELR and DivideMix) were found to amplify the final ascent. This means that when using these advanced techniques, choosing the right model size becomes even more critical for performance.
- Sample Size Mitigation: Increasing the number of training samples ($n$) reduces the noise-to-sample-size ratio, which effectively "pushes" the final ascent further out or eliminates it, confirming that more data is a primary cure for this scaling pathology.
Limitations and Future Work
- Random Masking vs. Structured Pruning: The study uses random parameter dropping to isolate the effect of density. The authors note that investigating how specific, non-random pruning techniques (like those based on gradients or magnitudes) interact with the final ascent is a promising future direction.
- Complexity Measures: While the paper explores spectral norms and trajectory lengths to explain function complexity, a more formal theoretical link between the "final ascent" and the recent theory of "benign vs. catastrophic overfitting" remains to be fully established.
- Architecture Specifics: While tested on ViTs and ResNets, the exact point where the final ascent begins is highly dependent on the specific architecture and noise type, making it hard to predict a "safe" model size without empirical testing.
Summary updated