Abstract
Inspired by the remarkable zero-shot generalization capacity of vision-language pre-trained models, we seek to leverage the supervision from the CLIP model to alleviate the burden of data labeling. However, such supervision inevitably contains label noise, which significantly degrades the discriminative power of the classification model. In this work, we propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. Firstly, a class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels and boost the tolerance to noisy labels. Secondly, ensemble labels are adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels. This framework effectively reduces the impact of noisy labels from the CLIP model by combining both techniques. Experiments on multiple benchmark datasets demonstrate substantial improvements over other state-of-the-art methods.
Introduction: The Challenge of Noisy Pseudo-Labels
Deep learning thrives on massive, accurately labeled datasets. However, manual labeling is a notorious bottleneck—it is expensive, slow, and often impractical for large-scale applications. Recently, OpenAI’s CLIP (Contrastive Language-Image Pretraining) has emerged as a potential solution, offering "zero-shot" classification by comparing images to text descriptions.
The primary drawback is that CLIP’s pseudo-labels contain significant, unbalanced noise. For instance, in the EuroSAT dataset, CLIP identifies "Residential" areas with 95% accuracy but fails on "Pasture," achieving only 1% accuracy. Standard training on these labels leads to confirmation bias, where the model memorizes noise rather than learning underlying patterns. Existing noise-robust methods often assume uniform noise distribution, an assumption that real-world CLIP performance contradicts.
Methodology: Transductive CLIP
The proposed Transductive CLIP framework approaches the noise problem through two primary mechanisms:
- Class-Conditional Contrastive Learning ($C^3L$): While standard contrastive learning ensures consistency in feature space, $C^3L$ operates in prediction space. It forces the model to produce consistent class predictions for different augmentations of the same image. Because clean labels are easier for a network to fit than random noise, this mechanism allows the "correct" signals to eventually dominate, pushing the model toward a robust decision boundary.
- Ensemble Labels: To stabilize training, the framework uses a temporal moving average for label updates. By combining the initial CLIP prediction with the average of all previous model predictions, this ensemble acts as a low-pass filter, suppressing fluctuations caused by noisy samples and preventing "negative updating."
The architecture utilizes a ViT-B/32 CLIP model as the "teacher" and a standard ResNet-50 as the "student" classification network.
Experimental Results
Transductive CLIP was evaluated on five diverse benchmarks: Caltech101, DTD, EuroSAT, NWPU-RESISC, and Flower102. The framework consistently outperformed both the raw CLIP model and several state-of-the-art noise-handling baselines, such as DivideMix and FixMatch.
As shown in the table above, the method achieved an average accuracy of 65.83%, representing a +10.84% gain over the baseline CLIP Model. The most dramatic improvement occurred on the NWPU-RESISC remote sensing dataset, where accuracy jumped by 17.17%. These results demonstrate that even when initial CLIP labels are poor for certain classes, the $C^3L$ loss allows the model to self-correct by leveraging the intrinsic structure of the data.
Conclusion and Future Work
Transductive CLIP provides a powerful way to leverage unlabeled data by treating CLIP as a noisy teacher rather than an infallible oracle. While effective, the current framework is transductive, requiring access to the entire unlabeled dataset during training. Future work may explore adapting this to an inductive setting and reducing the computational overhead associated with iterative ensemble label updates for massive datasets.
Summary updated