Abstract
Deep Neural Networks (DNNs) are prone to memorizing incorrect labels in noisy datasets, which degrades generalization. This paper introduces CoDis (Co-Discrepancy), a novel sample selection method that robustly trains dual networks by mining high-discrepancy examples. Unlike previous disagreement-based methods that suffer from sample inefficiency, CoDis utilizes continuous probability discrepancy to identify "hard" clean examples. Experimental results across synthetic and real-world benchmarks demonstrate that CoDis consistently outperforms state-of-the-art methods, particularly in high-noise and class-imbalanced scenarios.
Introduction
Training robust DNNs on datasets with noisy labels is a critical challenge in machine learning. While DNNs possess the capacity to fit arbitrary noise, they typically exhibit a "memorization effect," where simple, clean patterns are learned before the model begins to overfit noisy labels. Sample selection methods exploit this by identifying likely-clean samples early in training.
A common approach involves training two parallel networks to filter noise. Previous strategies, such as "Update by Disagreement," only train on samples where the two networks predict different labels. However, label disagreement is a restrictive condition that often leads to sample inefficiency, especially as the networks converge. CoDis addresses this by shifting the focus from discrete label disagreement to continuous probability discrepancy, allowing for a more nuanced and efficient selection of informative training data.
Methodology
CoDis employs a dual-network architecture ($f_1$ and $f_2$) with a cross-update training strategy. The core innovation lies in the Joint Selection Criterion, which balances classification accuracy with model divergence.
Discrepancy Loss
Instead of binary disagreement, CoDis measures the Jensen-Shannon (JS) divergence between the output probability distributions of the two networks. This discrepancy loss ($\mathcal{L}_D$) captures subtle differences in how each network perceives a sample, even if they agree on the final class label.
Joint Selection
Samples are selected for training based on a joint loss function: $$\mathcal{L}_J = \mathcal{L}_C - \alpha \cdot \mathcal{L}_D$$ where $\mathcal{L}_C$ is the standard cross-entropy loss and $\alpha$ is a hyperparameter. By minimizing $\mathcal{L}_J$, the algorithm selects samples that have a small classification loss (indicating they are likely clean) but high discrepancy (indicating they are informative and near the decision boundary). This process effectively mines "hard" clean examples that are crucial for defining sharp decision boundaries.
Experiments and Results
The authors evaluated CoDis on a diverse set of benchmarks, including MNIST, CIFAR-10/100, and large-scale real-world datasets like Clothing1M and WebVision. The method was tested against various noise types, including symmetric, pair-flip, and instance-dependent noise.
Performance and Stability
CoDis consistently achieved superior test accuracy compared to baselines like Co-teaching, Co-teaching+, and JoCor. For instance, on CIFAR-10 with 40% pair-flip noise, CoDis reached 73.12% accuracy, outperforming JoCor (71.62%). Notably, CoDis maintains high sample efficiency; on MNIST with 40% noise, it utilized ~60% of the data, whereas Co-teaching+ dropped to ~10%.
Robustness to Imbalance
Unlike methods that force network agreement (e.g., JoCor), which can collapse on long-tailed distributions, CoDis remains robust. In imbalanced noisy scenarios, CoDis outperformed competitors by over 5% in test accuracy, demonstrating its ability to maintain diverse "views" of the data without model collapse.
Conclusion
CoDis provides a robust framework for combating label noise by mining high-discrepancy examples. By utilizing continuous probability discrepancy, it overcomes the sample inefficiency of previous disagreement-based methods and excels at identifying hard, informative clean samples. Future work includes applying this principle to large-scale pre-training on noisy web-scraped data, where efficient sample selection is paramount.
Summary updated