Unlearning the Noisy Correspondence Makes CLIP More Robust

Haochen Han, Alex Jinpeng Wang, Peijun Ye, Fangming Liu

Abstract

Large-scale Vision-Language Models (VLMs) like CLIP rely on massive web-scraped datasets that are inherently noisy. This noise, termed Noisy Correspondence (NC), manifests as mismatched image-text pairs (False Positives) or semantically similar pairs treated as unrelated (False Negatives). Rather than the computationally expensive approach of robust pre-training from scratch, this paper introduces Noisy Correspondence Unlearning (NCU). NCU is a fine-tuning framework that identifies noisy samples in pre-trained models and uses "hardest negative" semantic information to surgically remove their influence. Experimental results demonstrate that NCU significantly boosts zero-shot accuracy and retrieval performance with minimal computational overhead.

The Challenge of Noisy Correspondence

Training models on billions of web-scraped pairs inevitably introduces errors. These errors generally fall into two categories:

  1. False Positives (FP): Mismatched pairs where the image and text are unrelated (e.g., a cat image paired with a dog caption).
  2. False Negatives (FN): Related samples that are treated as distinct negatives in the contrastive loss (e.g., two different images of the same breed of dog).
Illustration of False Positives and False Negatives in noisy image-text datasets.

Standard contrastive learning forces the model to align FPs and push apart FNs, leading to a distorted feature space. Existing solutions often focus on cleaning data before training, which is prohibitively expensive for large-scale models. NCU shifts the focus to "fixing" the model post-training.

The NCU Framework

The NCU framework operates through a three-stage fine-tuning process designed to refine the pre-trained CLIP feature space:

  1. Forget Set Identification: The model uses its own internal similarity scores to identify the most likely noisy pairs (the bottom $P\%$ of a batch), flagging them as False Positives.
  2. Negative Semantic Learning: To provide a clear direction for unlearning, the framework learns a "hardest negative" representation. Using learnable prompt vectors, it generates a textual representation that acts as the semantic opposite of the noisy caption (e.g., "Not a bird").
  3. Optimal Transport (OT) Unlearning: The fine-tuning is framed as an Optimal Transport problem. This allows the model to simultaneously push mismatched pairs apart (unlearning FPs) and allow for soft, many-to-many alignments between similar images and texts (mitigating FNs).
The NCU framework pipeline, utilizing Optimal Transport and negative semantic prompts to unlearn noisy correspondence.

Key Results and Insights

NCU demonstrates that it is more efficient to unlearn noise from a pre-trained model than to attempt to prevent it during initial training.

  • Performance Gains: NCU improved CLIP’s top-1 zero-shot accuracy on ImageNet by up to 4.0% and showed a massive 7.7% improvement in average recall on the Flickr30K retrieval task.
  • Efficiency: The method outperforms state-of-the-art robust training methods like SoftCLIP without the need for training from scratch or using external teacher models.
  • Geometric Integrity: By using "Text Relation Opposite" loss, NCU ensures that the unlearning process doesn't cause the feature space to collapse, maintaining the logical structure of embeddings.

Conclusion

NCU provides a scalable and efficient way to enhance the robustness of pre-trained VLMs. By treating noise as something to be "unlearned" rather than avoided, it offers a practical path forward for utilizing increasingly large and messy web-scale datasets. Future work will explore the application of NCU to even larger architectures and billion-scale datasets like LAION-5B.

Summary updated

BibTeX
@inproceedings{han2025unlearning,
  author = {Haochen Han and Alex Jinpeng Wang and Peijun Ye and Fangming Liu},
  title = {{Unlearning the Noisy Correspondence Makes CLIP More Robust}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure