Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise

Yihao Xue, Kyle Whitecross, Baharan Mirzasoleiman

TL;DR

This paper uncovers the "final ascent" phenomenon: in the presence of label noise, the generalization performance of overparameterized neural networks eventually degrades as model width or density increases. The authors demonstrate that while scaling up usually helps, high noise-to-sample-size ratios make intermediate-sized models optimal, and surprisingly, standard robust training methods can actually make this performance drop at large scales even worse.

What Problem Does This Paper Address?

The deep learning community has recently embraced the "double descent" phenomenon, which suggests that increasing model size beyond the point of interpolation continues to improve generalization, contradicting classical bias-variance trade-off intuition. However, most of these studies assume clean or low-noise labels.

In the real world, large-scale datasets often rely on noisy sources like web-crawling or crowd-sourcing. This paper investigates a critical gap: How does label noise interact with model scaling (width and density)? The authors challenge the "bigger is always better" mantra by showing that under sufficient noise, the test loss curve doesn't just descend—it starts climbing again.

Methods

The authors employ a two-pronged approach:

  1. Theoretical Analysis: They analyze random feature ridge regression in a high-dimensional asymptotic limit. By decomposing the expected risk (test loss) into Bias², Variance$_{clean}$, and Variance$_{noise}$, they mathematically characterize how label noise alters the shape of the variance curve.
  2. Model Density as a New Dimension: Beyond just model width, they introduce model density—the fraction of trainable weights—as a variable. They provide the first theoretical characterization of how reducing density (randomly dropping parameters) affects generalization under noise.
  3. Empirical Validation: They conduct extensive experiments across various architectures (MLPs, ResNets, Vision Transformers, and InceptionResNet-v2) to see if their theoretical findings hold in practical deep learning scenarios.

Datasets

The study uses a mix of synthetic and real-world noise:

  • MNIST: Handwritten digits with synthetic symmetric noise.
  • CIFAR-10 / CIFAR-100: Object recognition with both symmetric and asymmetric (class-dependent) synthetic noise.
  • Stanford Cars: A fine-grained classification dataset using "Web Noise"—real-world label errors introduced by automated web searches.

Key Insights and Ideas

  • The Final Ascent: The most striking finding is that the test loss curve can exhibit a U-shape or a "decreasing-increasing-decreasing-increasing" pattern. At very high widths, the noise-induced variance dominates, causing generalization to worsen.

    Figure 1: Risk decomposition showing the final ascent in test loss as width increases.
  • Variance Shape Transition: The authors show that label noise fundamentally changes the behavior of model variance. While clean variance is typically unimodal, noise-induced variance monotonically increases with width. At high noise levels, the total variance develops an "increasing-decreasing-increasing" shape.

    Figure 2: Decomposition of variance showing how noise-induced variance (dotted line) drives the final ascent.
  • The Regularization Paradox: Intuitively, one might expect stronger regularization ($\ell_2$ or weight decay) to mitigate noise. However, the authors find that stronger regularization actually exacerbates the final ascent, making the performance gap between intermediate and large models even wider.

    Figure 3: Stronger regularization (λ₀) makes the final ascent more pronounced.
  • Wider but Sparser is Better: The paper suggests that for a fixed parameter budget, a wider model with lower density (more sparsity) generalizes better under label noise than a narrower, dense model.

Key Results

  • Noise Thresholds: The final ascent can appear with as little as 20% label noise on datasets like CIFAR-10.
  • Density vs. Regularization: Adjusting model density can achieve lower test loss than simply tuning $\ell_2$ regularization. In experiments on Stanford Cars, reducing density significantly improved performance under 30% and 70% noise.
  • Robust Algorithms: State-of-the-art robust training methods (like ELR and DivideMix) were found to amplify the final ascent. This means that when using these advanced techniques, choosing the right model size becomes even more critical for performance.
  • Sample Size Mitigation: Increasing the number of training samples ($n$) reduces the noise-to-sample-size ratio, which effectively "pushes" the final ascent further out or eliminates it, confirming that more data is a primary cure for this scaling pathology.

Limitations and Future Work

  • Random Masking vs. Structured Pruning: The study uses random parameter dropping to isolate the effect of density. The authors note that investigating how specific, non-random pruning techniques (like those based on gradients or magnitudes) interact with the final ascent is a promising future direction.
  • Complexity Measures: While the paper explores spectral norms and trajectory lengths to explain function complexity, a more formal theoretical link between the "final ascent" and the recent theory of "benign vs. catastrophic overfitting" remains to be fully established.
  • Architecture Specifics: While tested on ViTs and ResNets, the exact point where the final ascent begins is highly dependent on the specific architecture and noise type, making it hard to predict a "safe" model size without empirical testing.

Summary updated

Abstract

Increasing the size of overparameterized neural networks has been a key in achieving state-of-the-art performance. This is captured by the double descent phenomenon, where the test loss follows a decreasing-increasing-decreasing pattern (or sometimes monotonically decreasing) as model width increases. However, the effect of label noise on the test loss curve has not been fully explored. In this work, we uncover an intriguing phenomenon where label noise leads to a \textit{final ascent} in the originally observed double descent curve. Specifically, under a sufficiently large noise-to-sample-size ratio, optimal generalization is achieved at intermediate widths. Through theoretical analysis, we attribute this phenomenon to the shape transition of test loss variance induced by label noise. Furthermore, we extend the final ascent phenomenon to model density and provide the first theoretical characterization showing that reducing density by randomly dropping trainable parameters improves generalization under label noise. We also thoroughly examine the roles of regularization and sample size. Surprisingly, we find that larger $\ell_2$ regularization and robust learning methods against label noise exacerbate the final ascent. We confirm the validity of our findings through extensive experiments on ReLu networks trained on MNIST, ResNets/ViTs trained on CIFAR-10/100, and InceptionResNet-v2 trained on Stanford Cars with real-world noisy labels.
BibTeX
@inproceedings{xue2024investigating,
  author = {Yihao Xue and Kyle Whitecross and Baharan Mirzasoleiman},
  title = {{Investigating the Impact of Model Width and Density on Generalization in Presence of Label Noise}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=QXCRgDfrzS},
}
Figure