Abstract
Measuring perceptual similarity is a key tool in computer vision. While metrics based on features from neural networks like CLIP are popular, they lack adversarial robustness. This paper demonstrates that adversarially robust CLIP models, termed R-CLIP$_F$, obtained through unsupervised adversarial fine-tuning, induce superior and robust perceptual metrics. These models outperform existing metrics in zero-shot settings and match state-of-the-art performance after fine-tuning. Furthermore, R-CLIP$_F$ enables robust NSFW content detection and exhibits enhanced interpretability through feature inversion, revealing rich visual concepts learned by the model.
Introduction
Perceptual metrics aim to quantify image similarity in a way that aligns with human judgment. While models like CLIP and DINO have set new standards, they are susceptible to adversarial perturbations—tiny, invisible changes that can drastically alter a model's output. This "adversarial brittleness" poses significant risks for sensitive applications such as NSFW content filtering and copyright protection. Contrary to the common belief of a robustness-accuracy trade-off, this work shows that adversarial training can actually improve clean performance in perceptual tasks.
Methodology
The researchers fine-tuned CLIP and DINO vision encoders using two primary adversarial training techniques:
- TeCoA: A supervised method that uses labels (like ImageNet) to ensure consistency under attack.
- FARE: An unsupervised method that minimizes the distance between clean and attacked image embeddings, preserving the original CLIP embedding space for easy integration.
Evaluations were conducted across diverse backbones, including Vision Transformers (ViTs) and ConvNeXt, in both zero-shot and fine-tuned (using LoRA) settings.
Performance and Robustness
The robust model R-CLIP$_F$ (based on ConvNeXt-B) achieves state-of-the-art zero-shot performance. On the NIGHTS dataset, it reached 90.6% clean accuracy and 74.3% robust accuracy, significantly outperforming non-robust counterparts like standard CLIP and DINO.
This improvement suggests that adversarial training provides a better inductive bias for human-like visual perception. Notably, the authors found that convolutional architectures like ConvNeXt often outperformed Transformers in these tasks, likely due to their spatial hierarchy.
Interpretability and Feature Inversion
Robust models exhibit "cleaner" gradients, which translates to better interpretability. Through feature inversion—maximizing cosine similarity to an image's embedding starting from noise—the authors showed that R-CLIP$_F$ reconstructs semantically recognizable images. In contrast, standard CLIP models produce only adversarial noise when inverted.
Applications: NSFW Content Detection
A critical application of robust metrics is in dataset filtering. In simulated attacks on NSFW filters, standard CLIP's detection accuracy plummeted to near zero. R-CLIP$_F$ maintained a 75% robust detection rate while preserving high accuracy on unperturbed images, proving its utility for secure content moderation and dataset cleaning.
Conclusion
Adversarial robustness is more than a security feature; it is a mechanism for aligning AI vision with human perception. R-CLIP models provide a robust, interpretable, and high-performing foundation for perceptual similarity tasks, challenging the traditional trade-off between security and accuracy. Future work includes scaling these metrics to massive retrieval pools and using them to guide generative models for better text-image alignment.
Summary updated