Introduction
Deep Neural Networks (DNNs) have achieved remarkable success in supervised learning, but their performance is heavily dependent on the quality of labeled training data. In real-world applications, datasets often contain "label noise"—incorrect labels resulting from crowdsourcing errors or automated data collection. Standard DNNs are prone to memorizing these noisy labels, which leads to overfitting and poor generalization. This research presents a robust training framework designed to mitigate these effects through a novel loss function, a cross-training strategy, and online data filtration.
The Three-Pronged Defense Against Noise
The proposed approach addresses the challenge of messy labels through three integrated components:
1. Rectified Cross-Entropy (RCE) Loss
Standard cross-entropy loss can be counterproductive when labels are incorrect, as it forces the model to minimize error even for garbage data. The authors introduce Rectified Cross-Entropy (RCE), which adaptively re-weights the loss. If the model's prediction for a label is very low—suggesting the label is likely wrong—the RCE loss assigns it a lower weight. This prevents the network from trying too hard to "learn" mislabeled samples.
2. Cross-Training Strategy
To prevent a single model from falling into a self-reinforcing loop of its own biases, the framework trains two identical models (Model A and Model B) simultaneously. These models engage in a "peer-review" process where Model A’s predictions are used to weight the loss for Model B, and vice versa. This "multi-view" approach ensures that the training curriculum for each model is guided by the independent perspective of its partner.
3. Online Data Filtration
In addition to re-weighting, the system actively cleans the training set. During training, the framework calculates the average prediction probability from both models. If this average falls below a specific threshold (typically $1/\text{number of classes}$), the sample is flagged as noisy and removed from the training set entirely, ensuring the models focus on high-quality data.
Experimental Results
The researchers evaluated the framework on several major benchmarks, including MNIST, CIFAR-10, CIFAR-100, and the real-world Clothing1M dataset.
As illustrated in the figure above, the framework demonstrates significant robustness across different noise scales. Even when the noise ratio ($\eta$) is as high as 0.8 (meaning 80% of labels are wrong), the model maintains a stable and useful accuracy level, avoiding the catastrophic performance drops seen in standard training methods.
Key Performance Highlights:
- CIFAR-100 (80% Noise): The proposed method achieved 37.8% accuracy, compared to just 10.1% for standard cross-entropy training.
- Clothing1M (Real-World Noise): The framework reached a Top-1 accuracy of 74.6%, outperforming the standard baseline of 68.9%. This confirms that the method is effective against complex, natural noise found in web-crawled data.
Conclusion
By combining adaptive loss weighting with a collaborative dual-model architecture, this research provides a powerful tool for training deep learning models in the "wild." While the dual-model approach increases computational overhead, the gains in robustness make it a compelling choice for large-scale applications where perfect labeling is impossible. Future work will focus on dynamic thresholding for data filtration to further refine the learning process.
Summary updated