Abstract
Memorization in over-parameterized neural networks could severely hurt generalization in the presence of mislabeled examples. However, mislabeled examples are hard to avoid in extremely large datasets collected with weak supervision. We address this problem by reasoning counterfactually about the loss distribution of examples with uniform random labels had they were trained with the real examples, and use this information to remove noisy examples from the training set. First, we observe that examples with uniform random labels have higher losses when trained with stochastic gradient descent under large learning rates. Then, we propose to model the loss distribution of the counterfactual examples using only the network parameters, which is able to model such examples with remarkable success. Finally, we propose to remove examples whose loss exceeds a certain quantile of the modeled loss distribution. This leads to On-the-fly Data Denoising (ODD), a simple yet effective algorithm that is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
Introduction
Training deep neural networks on massive datasets is the standard recipe for success in computer vision. However, the larger the dataset, the more likely it is to contain "noisy" or mislabeled examples. Because modern networks are over-parameterized, they are remarkably good at memorizing these mistakes, which ultimately cripples their ability to generalize to real-world data. Existing solutions often require complex "teacher" networks, prior knowledge of the noise rate, or expensive computations.
On-the-fly Data Denoising (ODD) addresses this by identifying and removing mislabeled data during the training process with almost zero extra computational cost. It leverages the fact that neural networks trained with large learning rates struggle to fit noisy labels, allowing for a clear separation between clean and corrupted data.
Methodology: On-the-fly Data Denoising (ODD)
ODD relies on two key observations regarding Stochastic Gradient Descent (SGD) and loss distributions. First, when training with a large learning rate (e.g., 0.1), SGD naturally favors clean examples; clean data converges to low loss quickly, while noisy data maintains a high loss. Second, to decide which examples are "too noisy," ODD models a "counterfactual" loss distribution: predicting what the loss would look like if labels were uniform random.
The ODD Workflow
- Warm-up Phase: Train the model on the full (noisy) dataset using a large learning rate for a set number of epochs.
- Counterfactual Modeling: Model the "noisy" loss distribution using only the weights of the network's final layer and a Gaussian assumption for the inputs. This allows the estimation of a threshold based on a chosen percentile.
- Pruning: Remove any example whose current loss exceeds the threshold.
- Final Training: Continue training to convergence on the remaining "clean" subset.
Surprisingly, the method does not require training on random labels to know what their loss would look like. By analyzing the final layer's weights, ODD can simulate the "noise floor" of the model, making it independent of the actual noise ratio in the dataset.
Experimental Results
ODD consistently outperformed standard Empirical Risk Minimization (ERM) and matched or beat complex state-of-the-art baselines across synthetic and real-world noise settings.
- CIFAR-100 (40% noise): ODD combined with mixup achieved 76.5% accuracy, compared to just 55.7% for standard training.
- Clothing1M: ODD achieved 73.5% accuracy on the noisy-only set, outperforming specialized methods like Joint Optimization and DMI.
- WebVision: ODD reached 72.6% Top-1 accuracy, proving it scales to millions of images without the massive computational overhead of methods requiring Jacobian regularization.
Beyond performance gains, ODD serves as an effective data auditing tool. When run on the nominally "clean" CIFAR-100 dataset, ODD flagged real mislabeled images that had escaped human review.
Conclusion
ODD provides a robust, scalable, and simple framework for training on noisy datasets. By modeling counterfactual loss distributions on-the-fly, it avoids the need for prior knowledge of noise rates or expensive auxiliary models. While it currently relies on a Gaussian assumption for final layer inputs and specific pruning schedules, its success on large-scale benchmarks like WebVision suggests it is a highly practical tool for modern deep learning pipelines.
Summary updated