DivideMix: Learning with Noisy Labels as Semi-supervised Learning

Junnan Li, Richard Socher, Steven C. H. Hoi

Introduction

Deep Neural Networks (DNNs) typically require massive, accurately labeled datasets to generalize effectively. However, high-quality human annotation is expensive and slow. While web-scraping offers a scalable alternative, it inevitably introduces "noisy labels"—incorrect tags that DNNs tend to memorize, leading to poor generalization. DivideMix addresses this by reframing the noisy label problem as a semi-supervised learning (SSL) task.

Methodology: Reframing Noise

DivideMix treats noisy samples as unlabeled data, leveraging SSL techniques to maintain robustness. The framework is built on three pillars:

1. Co-Divide

DNNs naturally learn clean patterns before memorizing noise. Consequently, clean samples typically exhibit lower loss than noisy ones. DivideMix fits a two-component Gaussian Mixture Model (GMM) to the per-sample loss distribution to dynamically partition data into a labeled "clean" set and an unlabeled "noisy" set.

2. Two-Network Co-Training

To mitigate confirmation bias—where a model reinforces its own initial errors—DivideMix trains two separate networks (Network A and Network B) simultaneously. Each network uses the data division predicted by the other, ensuring that errors from one network do not directly contaminate its own training process.

3. Improved MixMatch

Once divided, the data is processed using an enhanced MixMatch strategy:

  • Label Co-refinement: Labeled samples are refined by blending ground-truth labels with predictions from the peer network.
  • Label Co-guessing: Unlabeled samples receive "pseudo-labels" generated by averaging predictions from both networks.
  • MixUp: Synthetic examples are created by interpolating between image-label pairs, providing further regularization.
Overview of the DivideMix framework illustrating the Co-Divide process and two-network co-training strategy.

Key Results

The authors evaluated DivideMix on synthetic (CIFAR-10/100) and real-world (Clothing1M, WebVision) datasets. The framework demonstrated exceptional resilience to extreme noise levels.

Comparative performance of DivideMix against state-of-the-art methods on CIFAR-10 and CIFAR-100 across various noise ratios.

As shown in the table above, DivideMix significantly outperforms previous methods, particularly in high-noise regimes. On CIFAR-100 with 90% noise, it achieved 31.5% accuracy, a substantial lead over the next best method (~24%). In real-world benchmarks, it set new records on Clothing1M (74.76%) and WebVision (77.32% top-1 accuracy).

Discussion and Insights

DivideMix demonstrates that noisy labels need not be a liability. By stripping suspect labels and treating the images as unlabeled data, the model continues to learn valuable visual features. The "power of divergence" between the two networks, combined with entropy maximization during an initial warm-up phase to prevent premature commitment to noise, makes the system uniquely robust. While the framework introduces some computational overhead due to dual-network training, its performance gains in noisy environments offer a compelling trade-off for large-scale, real-world applications.

Summary updated

BibTeX
@inproceedings{li2020dividemix,
  author = {Junnan Li and Richard Socher and Steven C. H. Hoi},
  title = {{DivideMix: Learning with Noisy Labels as Semi-supervised Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=fyloJHmTVwh},
}
Figure