Abstract
This paper investigates the robustness of vision-language models (VLMs) against adversarial visual perturbations and introduces a novel "double visual defense" to enhance this robustness. Unlike previous approaches that resort to lightweight adversarial fine-tuning of a pre-trained CLIP model, we perform large-scale adversarial vision-language pre-training from scratch using web-scale data. We then strengthen the defense by incorporating adversarial visual instruction tuning. The resulting models, $\Delta$CLIP and $\Delta^2$LLaVA, show substantially enhanced zero-shot robustness and set a new state-of-the-art in adversarial defense for VLMs. For example, the adversarial robustness of $\Delta$CLIP surpasses that of the previous best models on ImageNet-1k by ~20%. Furthermore, our models exhibit stronger zero-shot recognition capability, fewer hallucinations, and superior reasoning performance compared to baselines. Project page: https://doublevisualdefense.github.io/
Introduction: The Vulnerability of Vision-Language Models
Vision-Language Models (VLMs) like CLIP and LLaVA are the engines behind modern AI's ability to perceive and reason about multimodal data. However, they suffer from a critical security flaw: adversarial vulnerability. Malicious actors can add subtle, imperceptible noise to an image that causes the model to hallucinate, misclassify content, or even follow dangerous hidden instructions.
Previous attempts to fix this usually involved "lightweight" fine-tuning on small datasets. While this makes models robust to specific data, it often breaks their general intelligence—a phenomenon known as the "robustness-accuracy trade-off." These models become specialized defenders but lose their ability to generalize across diverse visual domains.
The Double Visual Defense Framework
We propose a "Double Visual Defense" strategy that integrates robustness into the model's DNA across two distinct stages of training.
Stage 1: Adversarial Contrastive Pre-training ($\Delta$CLIP)
Instead of fine-tuning a pre-existing model, we trained a CLIP model from scratch on the massive DataComp-1B dataset (1 billion image-text pairs). During training, the model was constantly challenged by adversarial examples generated on-the-fly. This created a robust visual backbone that maintains high "clean" performance while generalizing across the entire visual manifold.
Stage 2: Adversarial Visual Instruction Tuning ($\Delta^2$LLaVA)
We then integrated this $\Delta$CLIP backbone into a LLaVA-style architecture. Crucially, we introduced a second layer of defense: adversarial visual instruction tuning. During the phase where the model learns to follow human instructions, we perturbed the visual inputs to force the model to remain grounded in the true image content rather than being swayed by adversarial noise.
Experimental Results and Key Insights
Our evaluation spanned over 20 datasets, covering classification, captioning, visual QA, and reasoning.
- State-of-the-Art Robustness: $\Delta$CLIP outperformed the previous best robust model (FARE) on ImageNet-1k by ~20%. On the Stanford Cars dataset, it achieved a ~70% absolute robustness improvement (a ~700% relative increase).
- VLM Task Performance: Compared to prior art, $\Delta^2$LLaVA showed a ~30% improvement in image captioning robustness and a ~20% improvement in Visual Question Answering (VQA).
- No "Clean" Penalty: Unlike previous robust models, $\Delta$CLIP remains competitive with the original OpenAI CLIP on standard benchmarks, effectively bridging the utility gap.
- Reduced Hallucination: One of the most important side effects of this training is a significant drop in hallucinations. Because the model is trained to ignore misleading visual noise, it becomes much more disciplined about reporting only what is actually in the image.
- Targeted Attack Defense: Under aggressive targeted attacks, $\Delta^2$LLaVA maintained an average Attack Success Rate (ASR) of only 3.3%, whereas standard models were completely compromised (100% ASR).
Emergent Phenomena: Typographic Attacks
As models become more robust to pixel-level noise, adversarial attacks naturally evolve into typographic attacks. The noise starts to look like human-readable text (e.g., the word "DOG" appearing faintly over a cat) to fool the model. This is a significant "aha moment," indicating that the model is finally "looking" at semantic features rather than just statistical textures. Future work should explicitly incorporate typographic-image-based training to address this new frontier.
Conclusion
The Double Visual Defense framework demonstrates that scale is the antidote to overfitting in adversarial training. By moving away from quick fixes and integrating robustness into the pre-training and instruction-tuning phases, we can create VLMs that are both highly capable and truly robust.
Summary updated