TL;DR
This paper introduces Sim-CLIP, an unsupervised method to make the CLIP vision encoder more robust to adversarial attacks without sacrificing its ability to understand image semantics. By using a Siamese network and cosine similarity, Sim-CLIP significantly improves the performance of Vision-Language Models (VLMs) on downstream tasks when faced with perturbed images, offering a practical solution for more secure multimodal AI.
What Problem Does This Paper Address?
Vision-Language Models (VLMs) are increasingly integrated into consumer technologies, enabling tasks like image captioning and visual question answering. These models often rely on powerful pre-trained vision encoders, with CLIP being a popular choice. However, these encoders are highly vulnerable to adversarial perturbations – subtle, often imperceptible changes to input images that can drastically alter the model's output, leading to incorrect or nonsensical results. This vulnerability poses a significant safety and reliability concern, especially for applications in sensitive domains. Existing defenses often struggle to maintain semantic fidelity or require computationally expensive training setups.
Methods
Sim-CLIP tackles this problem through an unsupervised adversarial fine-tuning approach for the CLIP vision encoder. The core idea is to train the encoder to produce similar representations for both clean and adversarially perturbed images.
- Siamese Architecture: Sim-CLIP employs a Siamese network structure. This means two identical CLIP encoders are used: one processing the clean image and the other processing the adversarially perturbed image.
- Cosine Similarity Loss: Instead of traditional L2 loss, Sim-CLIP uses cosine similarity between the embeddings of the clean and perturbed images. This is crucial because cosine similarity focuses on the direction of the embeddings, which is more indicative of semantic content, especially in high-dimensional spaces where L2 distances can become less discriminative.
- Symmetric Stop-Gradient: To prevent "loss collapse" (where the model learns trivial solutions) and avoid the need for large batch sizes or momentum encoders (common in other contrastive learning methods), Sim-CLIP uses a symmetric loss formulation with a stop-gradient mechanism. This ensures that gradients flow through one encoder at a time, promoting stable learning without excessive computational overhead.
- PGD Perturbation: Adversarial examples are generated using the Projected Gradient Descent (PGD) method, a standard technique for creating strong adversarial perturbations.
Datasets
- Training: ImageNet was used for unsupervised adversarial fine-tuning of the CLIP encoder.
- Evaluation:
- Image Captioning: COCO and Flickr30k datasets.
- Visual Question Answering (VQA): VizWiz and OKVQA datasets.
- Zero-shot Image Classification: CIFAR-10, CIFAR-100, EuroSAT, PCAM, and Flowers datasets.
Key Insights and Ideas
- Cosine Similarity for Semantics: The paper highlights that cosine similarity is a more effective metric than L2 distance for aligning high-dimensional embeddings in adversarial training, as it better captures semantic relationships by focusing on vector direction.
- Stop-Gradient for Efficiency: The symmetric stop-gradient mechanism is a clever way to achieve stable adversarial training and prevent loss collapse without the computational burden of large batches or momentum encoders, making the method more practical.
- Plug-and-Play Robustness: Sim-CLIP's fine-tuned encoder can be directly swapped into existing VLMs without requiring further VLM fine-tuning, significantly simplifying deployment and adoption.
- Trade-off Management: The work demonstrates that it's possible to improve adversarial robustness without a significant drop in clean accuracy, and in some cases, even improve it, challenging the common perception of this trade-off.
Key Results
- Untargeted Attacks: Sim-CLIP significantly outperforms existing robust CLIP variants (TeCoA, FARE) on image captioning (COCO, Flickr30k) and VQA (VizWiz, OKVQA) tasks under various untargeted adversarial attack strengths (up to € = 8/255). For instance, it achieved up to +7.6 CIDEr improvement on COCO.
- Targeted Attacks: Sim-CLIP reduces the success rate of targeted attacks from 100% for vanilla CLIP to 0% at € = 4/255, while generating higher-quality captions compared to other robust methods.
- Zero-Shot Classification: Sim-CLIP improves robust accuracy by an average of 3.4% over state-of-the-art robust CLIP models across multiple zero-shot classification benchmarks, showing strong generalization.
- Clean Accuracy: Sim-CLIP achieves the best clean performance among the robust models evaluated.
Limitations and Future Work
- Limitations: The paper acknowledges a slight degradation in clean performance for robust models compared to the original CLIP, though Sim-CLIP minimizes this. The primary focus is on the CLIP ViT-L/14 encoder, and its effectiveness on other encoder architectures is not explored.
- Future Work: The authors suggest that their work underscores the importance of reinforcing foundational models like CLIP. Future directions could involve extending this approach to other vision encoders or exploring more sophisticated adversarial attack scenarios. The plug-and-play nature of Sim-CLIP also opens avenues for its integration into a wider range of multimodal applications.
Summary updated