Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics

Francesco Croce, Christian Schlarmann, Naman Deep Singh, Matthias Hein

TL;DR

Researchers have discovered that adversarially training CLIP models doesn't just make them harder to hack—it actually makes them better at mimicking human visual perception. These robust models, dubbed R-CLIP, outperform standard models in identifying image similarity (even in "clean" settings) and provide a secure foundation for sensitive tasks like NSFW content filtering.

What Problem Does This Paper Address?

In computer vision, "perceptual metrics" are tools used to determine how similar two images look to a human. While deep learning models like CLIP and DINO have become the gold standard for this, they suffer from a major flaw: adversarial brittleness.

A malicious actor can add a tiny, invisible "perturbation" to an image that completely changes its digital fingerprint without changing how it looks to a human. This is a massive security risk for applications like:

  • NSFW Detection: An attacker could bypass filters by slightly altering a prohibited image.
  • Copyright Protection: Small tweaks could hide infringing content from automated scanners.
  • Dataset Filtering: Harmful content could be "camouflaged" to stay in training sets.

Furthermore, there is a long-standing belief in AI that there is a "robustness-accuracy trade-off"—meaning that making a model secure usually makes it less accurate. This paper challenges that notion in the context of perceptual similarity.

Methods

The authors fine-tune the vision encoders of CLIP and DINO models using two primary adversarial training techniques:

  1. TeCoA: A supervised method that uses labels (like ImageNet) to teach the model to remain consistent under attack.
  2. FARE: An unsupervised method that minimizes the distance between the embeddings of a "clean" image and its "attacked" version. This is particularly useful because it preserves the original CLIP embedding space, making it a "drop-in" replacement for existing workflows.

They tested various backbones, including Vision Transformers (ViTs) and ConvNeXt (a modern convolutional architecture). They evaluated these models in two ways: Zero-shot (using the model as-is) and Fine-tuned (using LoRA to adapt the model specifically to human similarity datasets).

Datasets

The study utilizes several key benchmarks to measure how well models align with human judgment:

  • NIGHTS & BAPPS: The primary datasets for "Two Alternatives Forced Choice" (2AFC) tasks, where a model must pick which of two images is more similar to a reference.
  • MS-COCO: Used for qualitative image-to-image retrieval.
  • ROxford & RParis: Standard benchmarks for landmark and image retrieval.
  • THINGS: A dataset containing image triplets to test categorical variations.
  • NSFW Public Dataset: A collection of safe vs. unsafe images used to simulate real-world content filtering attacks.

Key Insights and Ideas

  • The Reversed Trade-off: Surprisingly, for zero-shot perceptual metrics, adversarial training improves clean performance. Robust CLIP models were 3-7% more accurate on clean data than their non-robust counterparts.
  • Inductive Bias Matters: While the field has moved toward Transformers, the authors found that ConvNeXt (convolutional) often performed better as a perceptual metric, suggesting that the spatial hierarchy of convolutions is helpful for capturing human-like similarity.
  • Interpretability via Inversion: Robust models have "cleaner" gradients. The authors showed that you can "invert" a robust CLIP embedding to reconstruct a recognizable image of the original subject. Doing this with a standard CLIP model results in meaningless noise.
  • Generalization vs. Overfitting: Models specifically fine-tuned on similarity datasets (like DreamSim) often overfit and perform poorly on retrieval tasks. In contrast, zero-shot R-CLIP models generalize much better across different types of visual tasks.

Key Results

  • SOTA Performance: The robust model R-CLIPF (based on ConvNeXt-B) achieved state-of-the-art zero-shot performance on the NIGHTS dataset, reaching 90.6% clean accuracy and 74.3% robust accuracy.
  • NSFW Security: In a simulated attack on an NSFW filter, standard CLIP's detection accuracy plummeted to near zero. R-CLIPF maintained a 75% detection rate under the same attack while keeping high accuracy on unperturbed images.
  • Feature Inversion: The paper qualitatively demonstrates that R-CLIP "understands" high-level concepts better. When asked to visualize what the model "sees" for a prompt like "The Last Supper," the robust model generates a semantically correct layout, while the standard model fails.

Limitations and Future Work

The authors acknowledge a few areas for further exploration:

  • Retrieval Pool Size: The NSFW detection tests used a relatively small pool of 500 images; scaling this to billions of images (like the full LAION set) remains a challenge.
  • Architecture Scope: While they tested ViTs and ConvNeXt, they didn't explore every possible backbone or pre-training dataset.
  • Generative Guidance: A promising future direction is using these robust metrics to guide generative models (like Stable Diffusion) to ensure better alignment between text prompts and generated images.

In conclusion, this research suggests that adversarial robustness isn't just a defensive "shield"—it's a path toward building AI that actually sees the world more like we do.

Summary updated

Abstract

Measuring perceptual similarity is a key tool in computer vision. In recent years perceptual metrics based on features extracted from neural networks with large and diverse training sets, e.g. CLIP, have become popular. At the same time, the metrics extracted from features of neural networks are not adversarially robust. In this paper we show that adversarially robust CLIP models, called R-CLIPF, obtained by unsupervised adversarial fine-tuning induce a better and adversarially robust perceptual metric that outperforms existing metrics in a zero-shot setting, and further matches the performance of state-of-the-art metrics while being robust after fine-tuning. Moreover, our perceptual metric achieves strong performance on related tasks such as robust image-to-image retrieval, which becomes especially relevant when applied to “Not Safe for Work” (NSFW) content detection and dataset filtering. While standard perceptual metrics can be easily attacked by a small perturbation completely degrading NSFW detection, our robust perceptual metric maintains high accuracy under an attack while having similar performance for unperturbed images. Finally, perceptual metrics induced by robust CLIP models have higher interpretability: feature inversion can show which images are considered similar, while text inversion can find what images are associated to a given prompt. This also allows us to visualize the very rich visual concepts learned by a CLIP model, including memorized persons, paintings and complex queries.
BibTeX
@inproceedings{croce2025adversarially,
  author = {Francesco Croce and Christian Schlarmann and Naman Deep Singh and Matthias Hein},
  title = {{Adversarially Robust CLIP Models Can Induce Better (Robust) Perceptual Metrics}},
  booktitle = {SaTML},
  year = {2025},
  url = {https://openreview.net/forum?id=clAHe1yWMh},
}

Loading related content...

Figure