Combating Noisy Labels with Sample Selection by Mining High-Discrepancy Examples

Xiaobo Xia, Bo Han, Yibing Zhan, Jun Yu, Mingming Gong, Chen Gong, Tongliang Liu

Abstract

Deep Neural Networks (DNNs) are prone to memorizing incorrect labels in noisy datasets, which degrades generalization. This paper introduces CoDis (Co-Discrepancy), a novel sample selection method that robustly trains dual networks by mining high-discrepancy examples. Unlike previous disagreement-based methods that suffer from sample inefficiency, CoDis utilizes continuous probability discrepancy to identify "hard" clean examples. Experimental results across synthetic and real-world benchmarks demonstrate that CoDis consistently outperforms state-of-the-art methods, particularly in high-noise and class-imbalanced scenarios.

Introduction

Training robust DNNs on datasets with noisy labels is a critical challenge in machine learning. While DNNs possess the capacity to fit arbitrary noise, they typically exhibit a "memorization effect," where simple, clean patterns are learned before the model begins to overfit noisy labels. Sample selection methods exploit this by identifying likely-clean samples early in training.

A common approach involves training two parallel networks to filter noise. Previous strategies, such as "Update by Disagreement," only train on samples where the two networks predict different labels. However, label disagreement is a restrictive condition that often leads to sample inefficiency, especially as the networks converge. CoDis addresses this by shifting the focus from discrete label disagreement to continuous probability discrepancy, allowing for a more nuanced and efficient selection of informative training data.

Methodology

CoDis employs a dual-network architecture ($f_1$ and $f_2$) with a cross-update training strategy. The core innovation lies in the Joint Selection Criterion, which balances classification accuracy with model divergence.

Discrepancy Loss

Instead of binary disagreement, CoDis measures the Jensen-Shannon (JS) divergence between the output probability distributions of the two networks. This discrepancy loss ($\mathcal{L}_D$) captures subtle differences in how each network perceives a sample, even if they agree on the final class label.

Joint Selection

Samples are selected for training based on a joint loss function: $$\mathcal{L}_J = \mathcal{L}_C - \alpha \cdot \mathcal{L}_D$$ where $\mathcal{L}_C$ is the standard cross-entropy loss and $\alpha$ is a hyperparameter. By minimizing $\mathcal{L}_J$, the algorithm selects samples that have a small classification loss (indicating they are likely clean) but high discrepancy (indicating they are informative and near the decision boundary). This process effectively mines "hard" clean examples that are crucial for defining sharp decision boundaries.

Algorithm 1: The CoDis training procedure, illustrating the cross-update mechanism and joint sample selection.

Experiments and Results

The authors evaluated CoDis on a diverse set of benchmarks, including MNIST, CIFAR-10/100, and large-scale real-world datasets like Clothing1M and WebVision. The method was tested against various noise types, including symmetric, pair-flip, and instance-dependent noise.

Performance and Stability

CoDis consistently achieved superior test accuracy compared to baselines like Co-teaching, Co-teaching+, and JoCor. For instance, on CIFAR-10 with 40% pair-flip noise, CoDis reached 73.12% accuracy, outperforming JoCor (71.62%). Notably, CoDis maintains high sample efficiency; on MNIST with 40% noise, it utilized ~60% of the data, whereas Co-teaching+ dropped to ~10%.

Performance comparison of CoDis against state-of-the-art baselines across various asymmetric noise levels and datasets. CoDis (red line) shows superior stability and accuracy.

Robustness to Imbalance

Unlike methods that force network agreement (e.g., JoCor), which can collapse on long-tailed distributions, CoDis remains robust. In imbalanced noisy scenarios, CoDis outperformed competitors by over 5% in test accuracy, demonstrating its ability to maintain diverse "views" of the data without model collapse.

Conclusion

CoDis provides a robust framework for combating label noise by mining high-discrepancy examples. By utilizing continuous probability discrepancy, it overcomes the sample inefficiency of previous disagreement-based methods and excels at identifying hard, informative clean samples. Future work includes applying this principle to large-scale pre-training on noisy web-scraped data, where efficient sample selection is paramount.

Summary updated

Abstract

The sample selection approach is popular in learning with noisy labels. The state-of-the-art methods train two deep networks simultaneously for sample selection, which aims to employ their different learning abilities. To prevent two networks from converging to a consensus, their divergence should be maintained. Prior work presents that the divergence can be kept by locating the disagreement data on which the prediction labels of the two networks are different. However, this procedure is sample-inefficient for generalization, which means that only a few clean examples can be utilized in training. In this paper, to address the issue, we propose a simple yet effective method called CoDis. In particular, we select possibly clean data that simultaneously have high-discrepancy prediction probabilities between two networks. As selected data have high discrepancies in probabilities, the divergence of two networks can be maintained by training on such data. In addition, the condition of high discrepancies is milder than disagreement, which allows more data to be considered for training, and makes our method more sample-efficient. Moreover, we show that the proposed method enables to mine hard clean examples to help generalization. Empirical results show that CoDis is superior to multiple baselines in the robustness of trained models.
BibTeX
@inproceedings{xia2023combating,
  author = {Xiaobo Xia and Bo Han and Yibing Zhan and Jun Yu and Mingming Gong and Chen Gong and Tongliang Liu},
  title = {{Combating Noisy Labels with Sample Selection by Mining High-Discrepancy Examples}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00176},
}
Figure