Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics

Francesco Croce, Christian Schlarmann, Naman Deep Singh, Matthias Hein

Abstract

Measuring perceptual similarity is a key tool in computer vision. While metrics based on features from neural networks like CLIP are popular, they lack adversarial robustness. This paper demonstrates that adversarially robust CLIP models, termed R-CLIP$_F$, obtained through unsupervised adversarial fine-tuning, induce superior and robust perceptual metrics. These models outperform existing metrics in zero-shot settings and match state-of-the-art performance after fine-tuning. Furthermore, R-CLIP$_F$ enables robust NSFW content detection and exhibits enhanced interpretability through feature inversion, revealing rich visual concepts learned by the model.

Introduction

Perceptual metrics aim to quantify image similarity in a way that aligns with human judgment. While models like CLIP and DINO have set new standards, they are susceptible to adversarial perturbations—tiny, invisible changes that can drastically alter a model's output. This "adversarial brittleness" poses significant risks for sensitive applications such as NSFW content filtering and copyright protection. Contrary to the common belief of a robustness-accuracy trade-off, this work shows that adversarial training can actually improve clean performance in perceptual tasks.

Methodology

The researchers fine-tuned CLIP and DINO vision encoders using two primary adversarial training techniques:

  1. TeCoA: A supervised method that uses labels (like ImageNet) to ensure consistency under attack.
  2. FARE: An unsupervised method that minimizes the distance between clean and attacked image embeddings, preserving the original CLIP embedding space for easy integration.

Evaluations were conducted across diverse backbones, including Vision Transformers (ViTs) and ConvNeXt, in both zero-shot and fine-tuned (using LoRA) settings.

Performance and Robustness

The robust model R-CLIP$_F$ (based on ConvNeXt-B) achieves state-of-the-art zero-shot performance. On the NIGHTS dataset, it reached 90.6% clean accuracy and 74.3% robust accuracy, significantly outperforming non-robust counterparts like standard CLIP and DINO.

Clean and robust performance of perceptual metrics on the NIGHTS dataset.

This improvement suggests that adversarial training provides a better inductive bias for human-like visual perception. Notably, the authors found that convolutional architectures like ConvNeXt often outperformed Transformers in these tasks, likely due to their spatial hierarchy.

Interpretability and Feature Inversion

Robust models exhibit "cleaner" gradients, which translates to better interpretability. Through feature inversion—maximizing cosine similarity to an image's embedding starting from noise—the authors showed that R-CLIP$_F$ reconstructs semantically recognizable images. In contrast, standard CLIP models produce only adversarial noise when inverted.

Feature inversion comparison: R-CLIP$_F$ produces semantically correct reconstructions, while standard CLIP produces noise.

Applications: NSFW Content Detection

A critical application of robust metrics is in dataset filtering. In simulated attacks on NSFW filters, standard CLIP's detection accuracy plummeted to near zero. R-CLIP$_F$ maintained a 75% robust detection rate while preserving high accuracy on unperturbed images, proving its utility for secure content moderation and dataset cleaning.

Conclusion

Adversarial robustness is more than a security feature; it is a mechanism for aligning AI vision with human perception. R-CLIP models provide a robust, interpretable, and high-performing foundation for perceptual similarity tasks, challenging the traditional trade-off between security and accuracy. Future work includes scaling these metrics to massive retrieval pools and using them to guide generative models for better text-image alignment.

Summary updated

Abstract

Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse training sets, e.g. CLIP, have become popular. At the same time, the metrics extracted from features of neural networks are not adversarially robust. In this paper we show that adversarially robust CLIP models, called R-CLIP$_\textrm{F}$, obtained by unsupervised adversarial fine-tuning induce a better and adversarially robust perceptual metric that outperforms existing metrics in a zero-shot setting, and further matches the performance of state-of-the-art metrics while being robust after fine-tuning. Moreover, our perceptual metric achieves strong performance on related tasks such as robust image-to-image retrieval, which becomes especially relevant when applied to "Not Safe for Work" (NSFW) content detection and dataset filtering. While standard perceptual metrics can be easily attacked by a small perturbation completely degrading NSFW detection, our robust perceptual metric maintains high accuracy under an attack while having similar performance for unperturbed images. Finally, perceptual metrics induced by robust CLIP models have higher interpretability: feature inversion can show which images are considered similar, while text inversion can find what images are associated to a given prompt. This also allows us to visualize the very rich visual concepts learned by a CLIP model, including memorized persons, paintings and complex queries.
BibTeX
@inproceedings{croce2025adversarially,
  author = {Francesco Croce and Christian Schlarmann and Naman Deep Singh and Matthias Hein},
  title = {{Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics}},
  booktitle = {SaTML},
  year = {2025},
  url = {https://openreview.net/forum?id=clAHe1yWMh},
}
Figure