Transductive CLIP with Class-Conditional Contrastive Learning

Junchu Huang, Weijie Chen, Shicai Yang, Di Xie, Shiliang Pu, Yueting Zhuang

Abstract

Inspired by the remarkable zero-shot generalization capacity of vision-language pre-trained models, we seek to leverage the supervision from the CLIP model to alleviate the burden of data labeling. However, such supervision inevitably contains label noise, which significantly degrades the discriminative power of the classification model. In this work, we propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. Firstly, a class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels and boost the tolerance to noisy labels. Secondly, ensemble labels are adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels. This framework effectively reduces the impact of noisy labels from the CLIP model by combining both techniques. Experiments on multiple benchmark datasets demonstrate substantial improvements over other state-of-the-art methods.

Introduction: The Challenge of Noisy Pseudo-Labels

Deep learning thrives on massive, accurately labeled datasets. However, manual labeling is a notorious bottleneck—it is expensive, slow, and often impractical for large-scale applications. Recently, OpenAI’s CLIP (Contrastive Language-Image Pretraining) has emerged as a potential solution, offering "zero-shot" classification by comparing images to text descriptions.

The primary drawback is that CLIP’s pseudo-labels contain significant, unbalanced noise. For instance, in the EuroSAT dataset, CLIP identifies "Residential" areas with 95% accuracy but fails on "Pasture," achieving only 1% accuracy. Standard training on these labels leads to confirmation bias, where the model memorizes noise rather than learning underlying patterns. Existing noise-robust methods often assume uniform noise distribution, an assumption that real-world CLIP performance contradicts.

Figure 1: Confusion matrix illustrating CLIP's highly unbalanced noise distribution on the EuroSAT dataset.

Methodology: Transductive CLIP

The proposed Transductive CLIP framework approaches the noise problem through two primary mechanisms:

  1. Class-Conditional Contrastive Learning ($C^3L$): While standard contrastive learning ensures consistency in feature space, $C^3L$ operates in prediction space. It forces the model to produce consistent class predictions for different augmentations of the same image. Because clean labels are easier for a network to fit than random noise, this mechanism allows the "correct" signals to eventually dominate, pushing the model toward a robust decision boundary.
  2. Ensemble Labels: To stabilize training, the framework uses a temporal moving average for label updates. By combining the initial CLIP prediction with the average of all previous model predictions, this ensemble acts as a low-pass filter, suppressing fluctuations caused by noisy samples and preventing "negative updating."

The architecture utilizes a ViT-B/32 CLIP model as the "teacher" and a standard ResNet-50 as the "student" classification network.

Experimental Results

Transductive CLIP was evaluated on five diverse benchmarks: Caltech101, DTD, EuroSAT, NWPU-RESISC, and Flower102. The framework consistently outperformed both the raw CLIP model and several state-of-the-art noise-handling baselines, such as DivideMix and FixMatch.

Figure 6: Performance comparison of Transductive CLIP against state-of-the-art methods across five benchmark datasets.

As shown in the table above, the method achieved an average accuracy of 65.83%, representing a +10.84% gain over the baseline CLIP Model. The most dramatic improvement occurred on the NWPU-RESISC remote sensing dataset, where accuracy jumped by 17.17%. These results demonstrate that even when initial CLIP labels are poor for certain classes, the $C^3L$ loss allows the model to self-correct by leveraging the intrinsic structure of the data.

Conclusion and Future Work

Transductive CLIP provides a powerful way to leverage unlabeled data by treating CLIP as a noisy teacher rather than an infallible oracle. While effective, the current framework is transductive, requiring access to the entire unlabeled dataset during training. Future work may explore adapting this to an inductive setting and reducing the computational overhead associated with iterative ensemble label updates for massive datasets.

Summary updated

Abstract

Inspired by the remarkable zero-shot generalization capacity of vision-language pre-trained model, we seek to leverage the supervision from CLIP model to alleviate the burden of data labeling. However, such supervision inevitably contains the label noise, which significantly degrades the discriminative power of the classification model. In this work, we propose Transductive CLIP, a novel framework for learning a classification network with noisy labels from scratch. Firstly, a class-conditional contrastive learning mechanism is proposed to mitigate the reliance on pseudo labels and boost the tolerance to noisy labels. Secondly, ensemble labels is adopted as a pseudo label updating strategy to stabilize the training of deep neural networks with noisy labels. This framework can reduce the impact of noisy labels from CLIP model effectively by combining both techniques. Experiments on multiple benchmark datasets demonstrate the substantial improvements over other state-of-the-art methods.
BibTeX
@inproceedings{huang2022transductive,
  author = {Junchu Huang and Weijie Chen and Shicai Yang and Di Xie and Shiliang Pu and Yueting Zhuang},
  title = {{Transductive CLIP with Class-Conditional Contrastive Learning}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=cOHTEwjPWb},
}
Figure