Searching to Exploit Memorization Effect in Learning with Noisy Labels

Quanming Yao, Hansi Yang, Bo Han, Gang Niu, James Tin-Yau KwokICML 2020 · OpenReview

TL;DR

Deep neural networks tend to learn simple, clean patterns before they start memorizing noisy labels—a phenomenon known as the "memorization effect." This paper introduces S2E (Search to Exploit), an AutoML-based approach that automatically discovers the optimal schedule for selecting clean training samples. S2E outperforms manually tuned state-of-the-art methods in both accuracy and efficiency.

What Problem Does This Paper Address?

Training deep learning models on real-world data is often a "noisy" endeavor. Labels sourced from the internet or crowdsourcing are frequently incorrect. Because deep networks have massive learning capacities, they eventually overfit these incorrect labels, leading to poor generalization.

Test accuracy over time across different datasets and noise levels, illustrating the memorization effect.

Existing "sample selection" methods (like Co-teaching or MentorNet) try to combat this by only training on "small-loss" samples, assuming these are the clean ones. However, these methods rely on a manually designed schedule to decide how many samples to keep at each stage of training. These schedules are usually "one-size-fits-all" and don't adapt to the specific noise levels or characteristics of a new dataset. Manually tuning these schedules is tedious and often suboptimal.

Methods

The authors frame the search for the best sample selection schedule as an AutoML problem using bi-level optimization.

  1. Bi-level Formulation: The "outer" loop searches for the best selection function $R(t)$ using a clean validation set, while the "inner" loop trains the model parameters using the training set filtered by $R(t)$.
  2. Domain-Specific Search Space: Instead of searching through all possible functions, they define a compact search space based on four basis functions. These functions are designed to mimic the inverse of a typical learning curve (starting high, dropping as the model begins to memorize noise, and potentially rising slightly later).
  3. S2E Algorithm: To solve the optimization efficiently, they use stochastic relaxation. Instead of picking a single schedule, they maintain a probability distribution over the search space. They then use a Newton-based method—incorporating second-order information (the Hessian)—to update this distribution. This is significantly faster than standard derivative-free AutoML methods like Bayesian Optimization.

Datasets

The researchers evaluated their method on three standard image recognition benchmarks, injecting synthetic noise to test robustness: MNIST, CIFAR-10, and CIFAR-100.

They tested two types of noise: Symmetric flipping (labels are changed to any other class with equal probability) and Pair flipping (labels are swapped between similar classes), with noise levels ranging from 20% to 50%.

Key Insights and Ideas

  • The "Opposite" Shape Rule: The most effective selection schedules $R(t)$ generally have a shape opposite to the model's learning curve. When the model is learning fast (early on), you can trust more samples. When the model starts to plateau and overfit (later on), you must be more selective.
  • Dataset Sensitivity: The optimal schedule is highly dependent on the noise level. S2E automatically discovers that for noisier datasets, the model needs to drop more samples much earlier in the training process.
  • Hessian Efficiency: While many AutoML approaches use first-order gradients or random search, the authors show that calculating the Hessian for this specific bi-level problem is mathematically tractable and makes the search process much more efficient.

Key Results

  • Superior Accuracy: S2E consistently outperformed state-of-the-art methods like Co-teaching, Co-teaching+, and MentorNet across all datasets and noise types. For example, on CIFAR-100 with 50% noise, S2E achieved ~38.7% accuracy compared to Co-teaching’s ~36.5%.
Impact of different sample selection schedules on test accuracy.
  • Higher Label Precision: The samples selected by S2E were "cleaner" (contained a higher ratio of correctly labeled images) than those selected by competing methods.
  • Search Speed: S2E found the optimal hyperparameters with far fewer model evaluations than standard AutoML techniques like Bayesian Optimization (BO) or Hyperband, making it practical for real-world use.

Limitations and Future Work

  • Clean Validation Requirement: Like most AutoML approaches, S2E requires a small, clean validation set to guide the search. In some extreme real-world scenarios, even a small clean set might be hard to obtain.
  • Basis Function Constraints: While the four basis functions are expressive, the search is still confined to the "shape" they define. Future work could explore more flexible function approximators like small neural networks to define the schedule.
  • Computational Overhead: While more efficient than other AutoML methods, S2E still requires more computation than a single training run with a fixed, manual schedule.

Summary updated

Abstract

Sample selection approaches are popular in robust learning from noisy labels. However, how to properly control the selection process so that deep networks can benefit from the memorization effect i...
BibTeX
@inproceedings{yao2020searching,
  author = {Quanming Yao and Hansi Yang and Bo Han and Gang Niu and James Tin-Yau Kwok},
  title = {{Searching to Exploit Memorization Effect in Learning with Noisy Labels}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=saUlbVXxE7c},
}
Figure