Abstract
Training deep neural networks on datasets with noisy labels often leads to poor generalization due to overfitting on incorrect annotations. Traditional "small-loss" sample selection methods frequently suffer from confirmation bias, where the model reinforces its own errors. We present RankMatch, a framework that addresses these challenges through two novel components: Sample Selection via Confidence Voting (SCV) and Rank Contrastive Loss (RCL). SCV utilizes a committee-based approach with class prototypes to robustly identify clean samples, while RCL leverages the stability of feature channel rankings to learn consistent representations. Our experiments demonstrate that RankMatch achieves state-of-the-art performance, particularly in extreme noise scenarios, outperforming existing methods by significant margins.
Introduction: The Challenge of Noisy Labels
Deep learning's success is heavily dependent on large-scale, accurately labeled datasets. However, real-world data often contains "noisy" labels due to automated scraping or non-expert labeling. Standard approaches to Learning with Noisy Labels (LNL) typically rely on the "small-loss" hypothesis—the idea that models learn clean patterns before memorizing noise. Unfortunately, this one-dimensional metric can lead to a "confirmation bias" loop, where the model overfits to mislabeled samples it incorrectly identifies as clean, becoming increasingly confident in its own mistakes.
Methods
RankMatch introduces a multi-faceted approach to improve both sample selection and representation learning, moving beyond simple loss-based heuristics.
Sample Selection via Confidence Voting (SCV)
Instead of relying solely on loss values, RankMatch employs Confidence Voting. The model identifies a set of highly confident samples and uses K-means clustering to generate "prototypes" for each class. To determine if a sample is clean, the model evaluates its $k$-nearest prototypes. If these prototypes collectively "vote" for a class that matches the sample's original label, it is retained for training. This ensemble-like approach provides a much more stable signal than individual loss metrics, effectively filtering out noisy data.
Rank Contrastive Loss (RCL)
To ensure the model learns robust features despite label noise, we introduce Rank Contrastive Loss. While absolute feature values can be sensitive to noise, the relative importance (rank) of feature channels remains remarkably stable for similar semantic concepts. RCL identifies the top-$r$ most activated channels for an image. If two images share similar top-ranked channels, the model is encouraged to pull their representations together in the feature space, regardless of potentially incorrect labels.
Consistency Regularization
RankMatch further incorporates consistency regularization by enforcing that different augmentations of the same image (weakly augmented vs. strongly augmented) produce consistent predictions. This technique, borrowed from semi-supervised learning (e.g., FixMatch), helps the model focus on invariant semantic features rather than noise-induced artifacts.
Experimental Results
RankMatch was evaluated on standard benchmarks (CIFAR-10/100) and large-scale real-world datasets (Clothing1M, WebVision).
- Extreme Noise Robustness: On CIFAR-100 with 90% symmetric noise, RankMatch achieved 50.6% accuracy, a dramatic improvement over the previous state-of-the-art (DivideMix), which reached approximately 30%.
- Real-World Scalability: On Clothing1M, a dataset with 1 million noisy images, RankMatch reached 75.22% accuracy, surpassing recent methods like UNICON. On WebVision, it achieved a Top-1 accuracy of 79.91%, demonstrating its effectiveness on complex, web-crawled data.
Limitations and Future Work
While RankMatch demonstrates superior performance, it introduces additional computational overhead due to the K-means clustering and nearest-neighbor searches in the SCV module. Furthermore, the method requires a "warm-up" period to establish initial feature awareness, and its performance can be sensitive to the duration and handling of this phase. Future research could focus on optimizing these components for even greater efficiency and exploring automated hyperparameter tuning for the rank threshold $r$.
Conclusion
RankMatch shifts the paradigm of learning with noisy labels from simple loss-based selection to a more sophisticated framework based on confidence and feature consistency. By leveraging the stability of feature channel rankings and a robust voting-based selection process, RankMatch sets a new standard for training deep models in imperfect data environments.
Summary updated