Abstract
Not all data in a typical training set help with generalization; some samples can be overly ambiguous or outrightly mislabeled. This paper introduces a new method to identify such samples and mitigate their impact when training neural networks. At the heart of our algorithm is the Area Under the Margin (AUM) statistic, which exploits differences in the training dynamics of clean and mislabeled samples. A simple procedure - adding an extra class populated with purposefully mislabeled threshold samples - learns a AUM upper bound that isolates mislabeled data. This approach consistently improves upon prior work on synthetic and real-world datasets. On the WebVision50 classification task our method removes 17% of training data, yielding a 1.6% (absolute) improvement in test error. On CIFAR100 removing 13% of the data leads to a 1.2% drop in error.
Introduction
Modern deep learning relies on massive datasets, which are often "weakly-labeled" through automated scraping or crowdsourcing. This leads to the inclusion of mislabeled or highly ambiguous examples. While neural networks are robust to some noise, their capacity for memorization allows them to fit these incorrect labels, which can degrade generalization and hurt performance on unseen data. The authors propose a "plug-and-play" method to automatically identify and discard these harmful samples, allowing practitioners to clean their datasets without resorting to complex robust loss functions or multi-stage training pipelines.
Methodology
The proposed method identifies mislabeled data by analyzing training dynamics through the Area Under the Margin (AUM) statistic.
Area Under the Margin (AUM)
The margin for a sample at a given epoch is the difference between the logit (pre-softmax output) of its assigned class and the highest logit of any other class. The AUM is the average of this margin over all training epochs. Clean samples typically achieve high AUM as the network learns their features quickly. In contrast, mislabeled samples exhibit low or negative AUM because the network's learned features for the true visual class conflict with the assigned label, creating a "tug-of-war" during training.
Thresholding with Fake Classes
To determine a principled cutoff for AUM scores, the authors introduce "threshold samples." A small subset of training data is assigned to a newly created, artificial class. Since this class has no visual coherence, the network must rely on pure memorization to learn it. The AUM distribution of these threshold samples provides a baseline for pure memorization; samples with AUM scores below the 99th percentile of this distribution are flagged as mislabeled and removed.
Experimental Results
The AUM method was evaluated across synthetic noise settings and real-world weakly-labeled benchmarks.
Performance on Noisy Data
On datasets like CIFAR and Tiny ImageNet with synthetic label noise, AUM achieved over 90% precision and recall in identifying corrupted samples. In real-world scenarios like WebVision50, removing 17.8% of the data identified as mislabeled led to a 1.6% absolute improvement in test error (dropping from 21.4% to 19.8%).
Cleaning "Clean" Datasets
Surprisingly, the method also improves performance on standard benchmarks. Removing the bottom 13% of AUM-ranked samples from the standard CIFAR100 dataset resulted in a 1.2% drop in test error. This suggests that even highly curated datasets contain enough ambiguity or latent errors to actively hinder generalization. Notably, the AUM ranking is highly stable, showing >98% correlation across different architectures like ResNet and DenseNet, indicating it captures fundamental dataset properties rather than model quirks.
Conclusion
The Area Under the Margin ranking offers a simple, architecture-agnostic way to clean datasets. By focusing on training dynamics rather than static metrics, it effectively separates hard-but-useful examples from harmful mislabeled ones. This approach allows for significant gains in model generalization without the need for specialized training pipelines or architectural changes.
Summary updated