Abstract
Training deep neural networks on large-scale datasets often involves dealing with "noisy" or incorrectly labeled data, which typically degrades model performance through memorization of errors. This paper presents LaCoL (Latent Contrastive Learning), a novel framework that transforms label noise from a liability into a learning opportunity. By leveraging "negative correlations"—the high-confidence knowledge of what an image is not—LaCoL constructs robust representations in a latent metric space. Experimental results on synthetic and real-world noisy datasets, including CIFAR and Clothing1M, demonstrate that LaCoL significantly outperforms existing state-of-the-art methods by effectively utilizing the information inherent in noisy samples.
Introduction: The Challenge of Label Noise
Deep learning's success is heavily dependent on massive, accurately labeled datasets. However, obtaining high-quality human annotations at scale is prohibitively expensive. Many real-world datasets are instead curated via web scraping or crowdsourcing, leading to significant "label noise"—for instance, an image of a cat incorrectly labeled as a "dog."
Standard training objectives, such as Cross-Entropy loss, are highly sensitive to these errors. Models tend to "memorize" the incorrect labels, which destroys their ability to generalize to unseen, correctly labeled data. Traditional solutions often involve filtering out suspicious samples or attempting to guess the correct label (relabeling). However, filtering wastes potentially valuable data, and relabeling can lead to "confirmation bias," where the model reinforces its own initial mistakes.
The authors of this paper propose a paradigm shift: even a noisy sample contains useful information. While we might be uncertain if a mislabeled image is a "dog," we can be highly certain it is not a "car" or a "bird." This "negative information" provides a much more reliable signal for training in noisy environments.
Methodology: Latent Contrastive Learning (LaCoL)
The proposed LaCoL framework operates across two distinct spaces: the Label Space (for class prediction) and the Metric Space (for learning feature embeddings). The method is built upon three core pillars:
1. Divergence-Preserving Classification
To prevent the model from overfitting to noise, LaCoL employs a dual-augmentation strategy. It uses "weak" augmentations (simple flips and crops) to identify high-confidence samples. Subsequently, "strong" augmentations (complex transformations like RandAugment) are applied to these samples for the classification task. This approach encourages the model to learn robust, invariant features rather than memorizing specific pixel patterns associated with noisy labels.
2. Weakly-Supervised Latent Contrastive Learning
This is the central innovation of the paper. For every image, the model randomly selects $K$ images from other categories to serve as negative pairs. Instead of forcing the model to map an image to a single (potentially incorrect) positive label, LaCoL uses a contrastive loss to push the image's embedding away from these negative samples in a latent metric space.
This "pairwise negative correlation" is inherently more robust to noise. In a dataset with 100 classes, if a label is wrong, there is only a 1/99 chance that a randomly chosen "complementary label" (stating it is not class X) is also incorrect.
3. Cross-Space Similarity Consistency
To maintain alignment between the classification and embedding tasks, the authors introduce a regularization term. This term ensures that the similarity between images in the label space (based on predicted class probabilities) matches their similarity in the metric space (based on embedding proximity). This prevents the two learning objectives from drifting apart and ensures a cohesive representation.
Key Results
LaCoL was evaluated against state-of-the-art (SOTA) methods like Co-teaching, JoCoR, and JNPL across several benchmarks:
- CIFAR-10 & CIFAR-100: Under 40% symmetric noise, LaCoL achieved 92.33% accuracy on CIFAR-10, compared to just 51.15% for standard training. On CIFAR-100 with 40% instance-dependent noise, it reached 63.87% accuracy, significantly outperforming the baseline (42.26%).
- Clothing1M: On this large-scale real-world dataset (approx. 38.5% noise), LaCoL set a new benchmark with a top accuracy of 74.68%.
Conclusion
LaCoL demonstrates that label noise does not have to be purely detrimental. By shifting the focus from potentially incorrect positive labels to highly certain negative correlations, the framework achieves superior robustness and representation quality. While the method introduces some computational overhead due to dual augmentations and embedding heads, its performance on real-world noisy data like Clothing1M highlights its practical value.
Summary updated