SLADE: Shielding against Dual Exploits in Large Vision-Language Models

Md. Zarif Hossain, Ahmed ImteajCVPR 2025 · OpenReview

Abstract

Large Vision-Language Models (LVLMs) have emerged as transformative tools in multimodal tasks, seamlessly integrating pretrained vision encoders to align visual and textual modalities. Prior works have highlighted the susceptibility of LVLMs to dual exploits (gradient-based and optimization-based jailbreak attacks), which leverage the expanded attack surface introduced by the image modality. Despite advancements in enhancing robustness, existing methods fall short in their ability to defend against dual exploits while preserving fine-grained semantic details and overall semantic coherence under intense adversarial perturbations. To bridge this gap, we introduce SheiLding against Dual Exploits (SLADE), a novel unsupervised adversarial fine-tuning scheme that enhances the resilience of CLIP-based vision encoders. SLADE's dual-level contrastive learning approach balances the granular and the holistic, capturing fine-grained image details without losing sight of high-level semantic coherence. Extensive experiments demonstrate that SLADE-equipped LVLMs set a new benchmark for robustness against dual exploits while preserving fine-grained semantic details of perturbed images. Notably, SLADE achieves these results without compromising the core functionalities of LVLMs, such as instruction following, or requiring the computational overhead (e.g., large batch sizes, momentum encoders) commonly associated with traditional contrastive learning methods.

Introduction: The Vulnerability of Multimodal Models

Large Vision-Language Models (LVLMs) like LLaVA and OpenFlamingo integrate pretrained vision encoders (typically CLIP) to bridge visual and textual modalities. However, this integration introduces a significant attack surface. This paper identifies "dual exploits" as a primary threat: gradient-based attacks that perturb pixels to inject malicious strings, and optimization-based jailbreaks that bypass safety filters to generate harmful content.

Figure 1: Comparison of dual exploits on standard CLIP vs. SLADE-enhanced encoders.

Existing defenses often face a "robustness tax," where increased security degrades the model's ability to perceive fine-grained details. SLADE (SheiLding against Dual Exploits) addresses this by enhancing the resilience of vision encoders without sacrificing semantic precision or requiring massive computational resources.

Methodology: Dual-Level Contrastive Learning

SLADE employs an unsupervised adversarial fine-tuning scheme focused solely on the vision encoder. The core innovation is a dual-level contrastive learning approach that balances local and global features.

Figure 2: The SLADE framework featuring dual-level contrastive learning and a stop-gradient mechanism.

Key Technical Innovations:

  • Patch-Level Loss: By aligning embeddings of clean image patches with their adversarial counterparts, SLADE forces the model to retain granular spatial details despite noise. This allows the model to "see through" perturbations.
  • Global-Level Loss: This ensures high-level semantic coherence by aligning the full image embeddings of clean and adversarial pairs.
  • Stop-Gradient Mechanism: Inspired by SimSiam, this prevents feature collapse and enables efficient training with small batch sizes (e.g., 64), avoiding the need for memory-heavy momentum encoders.
  • Plug-and-Play Adoption: Once fine-tuned, the SLADE-enhanced CLIP encoder can be swapped into existing LVLMs without retraining the language model or projection layers.

Experimental Evaluation

The vision encoder was fine-tuned on unlabeled ImageNet data and evaluated across diverse tasks including captioning, VQA, zero-shot classification, and safety reasoning using benchmarks like AdvBench and VisIT-Bench.

Key Results

SLADE-equipped LVLMs significantly outperform standard models and prior defenses:

  • Defeating Targeted Attacks: In targeted string injection attacks where standard CLIP-based LLaVA had a 100% success rate for attackers, SLADE reduced this to 0% while maintaining high caption quality.
  • Jailbreak Resilience: Under VisualAdv attacks, toxicity rates dropped from 30.4% (standard LLaVA-13B) to 13.7% with SLADE, outperforming other state-of-the-art defenses like TeCoA.
  • Performance Preservation: SLADE maintains near-identical performance on clean data for reasoning and instruction following, proving that robustness does not necessitate a loss in utility.

Conclusion

SLADE demonstrates that a robust vision encoder is critical for grounding LLMs and preventing them from being led astray by adversarial visual inputs. By focusing on both local patches and global context, SLADE provides a scalable, efficient defense against dual exploits while preserving the fine-grained perception necessary for high-performance multimodal AI.

Summary updated

Abstract

Large Vision-Language Models (LVLMs) have emerged as transformative tools in multimodal tasks, seamlessly integrating pretrained vision encoders to align visual and textual modalities. Prior works have highlighted the susceptibility of LVLMs to dual exploits (gradient-based and optimization-based jailbreak attacks), which leverage the expanded attack surface introduced by the image modality. Despite advancements in enhancing robustness, existing methods fall short in their ability to defend against dual exploits while preserving fine-grained semantic details and overall semantic coherence under intense adversarial perturbations. To bridge this gap, we introduce SheiLding against Dual Exploits (SLADE), a novel unsupervised adversarial fine-tuning scheme that enhances the resilience of CLIP-based vision encoders. SLADE's dual-level contrastive learning approach balances the granular and the holistic, capturing fine-grained image details without losing sight of high-level semantic coherence. Extensive experiments demonstrate that SLADE-equipped LVLMs set a new benchmark for robustness against dual exploits while preserving fine-grained semantic details of perturbed images. Notably, SLADE achieves these results without compromising the core functionalities of LVLMs, such as instruction following, or requiring the computational overhead (e.g., large batch sizes, momentum encoders) commonly associated with traditional contrastive learning methods.
BibTeX
@inproceedings{hossain2025slade,
  author = {Md. Zarif Hossain and Ahmed Imteaj},
  title = {{SLADE: Shielding against Dual Exploits in Large Vision-Language Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=4wKfA5lGHF},
}
Figure