RankMatch: Fostering Confidence and Consistency in Learning with Noisy Labels

Ziyi Zhang, Weikai Chen, Chaowei Fang, Zhen Li, Lechao Chen, Liang Lin, Guanbin LiICCV 2023 · DOI

Abstract

Training deep neural networks on datasets with noisy labels often leads to poor generalization due to overfitting on incorrect annotations. Traditional "small-loss" sample selection methods frequently suffer from confirmation bias, where the model reinforces its own errors. We present RankMatch, a framework that addresses these challenges through two novel components: Sample Selection via Confidence Voting (SCV) and Rank Contrastive Loss (RCL). SCV utilizes a committee-based approach with class prototypes to robustly identify clean samples, while RCL leverages the stability of feature channel rankings to learn consistent representations. Our experiments demonstrate that RankMatch achieves state-of-the-art performance, particularly in extreme noise scenarios, outperforming existing methods by significant margins.

Introduction: The Challenge of Noisy Labels

Deep learning's success is heavily dependent on large-scale, accurately labeled datasets. However, real-world data often contains "noisy" labels due to automated scraping or non-expert labeling. Standard approaches to Learning with Noisy Labels (LNL) typically rely on the "small-loss" hypothesis—the idea that models learn clean patterns before memorizing noise. Unfortunately, this one-dimensional metric can lead to a "confirmation bias" loop, where the model overfits to mislabeled samples it incorrectly identifies as clean, becoming increasingly confident in its own mistakes.

Methods

RankMatch introduces a multi-faceted approach to improve both sample selection and representation learning, moving beyond simple loss-based heuristics.

Sample Selection via Confidence Voting (SCV)

Instead of relying solely on loss values, RankMatch employs Confidence Voting. The model identifies a set of highly confident samples and uses K-means clustering to generate "prototypes" for each class. To determine if a sample is clean, the model evaluates its $k$-nearest prototypes. If these prototypes collectively "vote" for a class that matches the sample's original label, it is retained for training. This ensemble-like approach provides a much more stable signal than individual loss metrics, effectively filtering out noisy data.

Overview of the Sample Selection via Confidence Voting (SCV) mechanism.

Rank Contrastive Loss (RCL)

To ensure the model learns robust features despite label noise, we introduce Rank Contrastive Loss. While absolute feature values can be sensitive to noise, the relative importance (rank) of feature channels remains remarkably stable for similar semantic concepts. RCL identifies the top-$r$ most activated channels for an image. If two images share similar top-ranked channels, the model is encouraged to pull their representations together in the feature space, regardless of potentially incorrect labels.

The RankMatch training pipeline and the detailed mechanism of Rank Contrastive Loss (RCL).

Consistency Regularization

RankMatch further incorporates consistency regularization by enforcing that different augmentations of the same image (weakly augmented vs. strongly augmented) produce consistent predictions. This technique, borrowed from semi-supervised learning (e.g., FixMatch), helps the model focus on invariant semantic features rather than noise-induced artifacts.

Experimental Results

RankMatch was evaluated on standard benchmarks (CIFAR-10/100) and large-scale real-world datasets (Clothing1M, WebVision).

  • Extreme Noise Robustness: On CIFAR-100 with 90% symmetric noise, RankMatch achieved 50.6% accuracy, a dramatic improvement over the previous state-of-the-art (DivideMix), which reached approximately 30%.
  • Real-World Scalability: On Clothing1M, a dataset with 1 million noisy images, RankMatch reached 75.22% accuracy, surpassing recent methods like UNICON. On WebVision, it achieved a Top-1 accuracy of 79.91%, demonstrating its effectiveness on complex, web-crawled data.

Limitations and Future Work

While RankMatch demonstrates superior performance, it introduces additional computational overhead due to the K-means clustering and nearest-neighbor searches in the SCV module. Furthermore, the method requires a "warm-up" period to establish initial feature awareness, and its performance can be sensitive to the duration and handling of this phase. Future research could focus on optimizing these components for even greater efficiency and exploring automated hyperparameter tuning for the rank threshold $r$.

Conclusion

RankMatch shifts the paradigm of learning with noisy labels from simple loss-based selection to a more sophisticated framework based on confidence and feature consistency. By leveraging the stability of feature channel rankings and a robust voting-based selection process, RankMatch sets a new standard for training deep models in imperfect data environments.

Summary updated

BibTeX
@inproceedings{zhang2023rankmatch,
  author = {Ziyi Zhang and Weikai Chen and Chaowei Fang and Zhen Li and Lechao Chen and Liang Lin and Guanbin Li},
  title = {{RankMatch: Fostering Confidence and Consistency in Learning with Noisy Labels}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00158},
}
Figure