Synergistic Network Learning and Label Correction for Noise-Robust Image Classification

Xin Wang, Chen Gong, Kong Bin, Eric J. Seibel, Xin Wang 0045, Youbing Yin, Qi Song

TL;DR

This paper introduces a novel framework that simultaneously learns network parameters and corrects noisy labels in image classification datasets. By iteratively training two networks and using their agreement to identify and fix incorrect labels, the method achieves state-of-the-art performance on challenging noisy datasets, outperforming existing approaches.

What Problem Does This Paper Address?

Deep learning models have revolutionized computer vision, but their success hinges on massive, meticulously labeled datasets. The reality is that collecting such datasets is expensive and time-consuming, leading to the widespread use of datasets with inaccurate or incorrect labels. This "label noise" can cause deep neural networks (DNNs) to overfit to these errors, significantly degrading their performance. While methods like small loss selection (which assumes DNNs learn general patterns before memorizing noise) and label correction exist, they often struggle to be effective simultaneously or require clean datasets. This paper tackles the challenge of building robust image classifiers in the presence of significant label noise, aiming to improve model performance without relying on perfectly clean data.

Methods

The core of this paper's contribution is a synergistic network learning and label correction framework. This approach operates iteratively, involving two main steps:

  1. Joint Training with Small Loss Selection: Two identical neural networks are trained simultaneously. They are encouraged to agree on predictions through an "agreement strategy" and a joint loss function that combines supervised learning loss and an agreement loss (Jensen-Shannon divergence). This joint loss helps identify samples where the networks are likely to be correct. During training, only samples with a "small loss" (indicating higher confidence) are used for backpropagation, guided by a selection rate $R(t)$.
  2. Label Correction: After training robust models with small loss selection, the framework identifies and corrects noisy labels. It leverages the assumption that when two networks agree, they are likely correct. Samples with low agreement loss are considered "confident," and among these, those with high supervised learning loss (indicating a potential mislabel) are selected for correction. The labels of these selected samples are updated based on the networks' consensus predictions.

This process is repeated iteratively. The authors also introduce a label correction rate $T_k$ that decreases over time and a retraining threshold $C_{restart}$ to trigger a full retraining of the networks when a significant portion of labels have been corrected.

Datasets

The proposed method was evaluated on the following datasets:

  • CIFAR-10: A standard benchmark dataset for image classification, corrupted with synthetic symmetric and asymmetric noise at various rates (20%, 50%, 80% for symmetric; 40% for asymmetric).
  • CIFAR-100: Similar to CIFAR-10 but with 100 classes, making it a more challenging classification task. It was also corrupted with synthetic noise.
  • Clothing1M: A large-scale, real-world dataset known for its significant label noise (estimated at 40%). The authors used the 1M images with noisy labels for training and the clean 14k/10k sets for validation/testing.

Key Insights and Ideas

  • Synergy is Key: The paper strongly advocates for the combined approach of learning network parameters and correcting labels iteratively. Neither small loss selection nor label correction alone is as effective as their synergistic combination.
  • Agreement as a Signal: The core idea that "different models are unlikely to agree on incorrect labels" is powerfully leveraged. The agreement between two networks serves as a robust indicator of label correctness.
  • Iterative Refinement: The iterative nature of the framework allows for a gradual improvement of both the model and the dataset. As the model becomes more accurate, its ability to identify and correct noisy labels also improves.
  • Robustness to High Noise: The method demonstrates significant improvements, especially in high noise scenarios (e.g., 80% symmetric noise), where many other methods fail.
  • Importance of Stable Training: The ablation study highlights that training robust models before each label correction step (intermittent updates) is more effective than continuous label updates. This allows the networks to stabilize and learn more reliable patterns before attempting to correct labels.

Key Results

The proposed method achieved state-of-the-art (SOTA) performance across all tested datasets and noise configurations:

  • CIFAR-10: Achieved the highest test accuracy, notably outperforming baselines by a significant margin (e.g., +6.21% at 50% symmetric noise). It also demonstrated a much smaller standard deviation in accuracy, indicating greater stability.
  • CIFAR-100: Again, the method was the overall accuracy winner, showing advantages of +1.6-1.9 over JoCoR in challenging symmetry settings.
  • Clothing1M: Achieved SOTA performance on this real-world noisy dataset, demonstrating its effectiveness in handling complex, unknown noise structures.
  • Ablation Study: Showed that using label update intervals (training for 50 epochs before correction) led to better test accuracy and a higher number of correct labels compared to continuous updates. Retraining after correction also improved performance.

Limitations and Future Work

  • Stopping Training Early: The authors note that for the Clothing1M dataset, their method's performance slightly decreased in the last epoch due to overfitting. They suggest that stopping training early based on validation set performance could improve accuracy.
  • Computational Cost: While not explicitly detailed as a limitation, iterative training and the use of two networks inherently increase computational requirements compared to single-network approaches.
  • Future Work: The paper implies that the backbone network architecture is interchangeable, suggesting future work could explore different network architectures within this framework. The robustness to different types and levels of noise is also a continuous area of research.

Summary updated

Abstract

Large training datasets almost always contain examples with inaccurate or incorrect labels. Deep Neural Networks (DNNs) tend to overfit training label noise, resulting in poorer model performance in practice. To address this problem, we propose a robust label correction framework combining the ideas of small loss selection and noise correction, which learns network parameters and reassigns ground truth labels iteratively. Taking the expertise of DNNs to learn meaningful patterns before fitting noise, our framework first trains two networks over the current dataset with small loss selection. Based on the classification loss and agreement loss of two networks, we can measure the confidence of training data. More and more confident samples are selected for label correction during the learning process. We demonstrate our method on both synthetic and real-world datasets with different noise types and rates, including CIFAR-10, CIFAR-100 and Clothing1M, where our method outperforms the baseline approaches.
BibTeX
@inproceedings{wang2022synergistic,
  author = {Xin Wang and Chen Gong and Kong Bin and Eric J. Seibel and Xin Wang and Youbing Yin and Qi Song},
  title = {{Synergistic Network Learning and Label Correction for Noise-Robust Image Classification}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9747470},
}
Figure