When a Large Language Model (LLM) undergoes safety alignment, it learns to reject harmful text prompts. However, when we equip that same aligned LLM with a vision encoder to create a Vision-Language Model (VLM), its safety guardrails frequently collapse.
This vulnerability is not merely a failure to filter bad images; it represents a fundamental safety evaluation gap. Current evaluation frameworks, largely inherited from text-only NLP or standard computer vision benchmarks, fail to capture the unique, emergent failure modes that occur when text and vision interact.
The Compositional Blind Spot
Why does a VLM fail even when its text and image inputs appear completely benign? The core issue is cross-modal compositionality. Traditional safety filters look at text and images in isolation. However, as demonstrated by Youngwan Lee et al., a critical blind spot exists in the "safe image + safe text $\rightarrow$ unsafe content" configuration.
Imagine an image of a standard household chemical and a text prompt asking for a simple cleaning instruction. Individually, both inputs are entirely safe. But when combined, they might inadvertently provide a recipe for a dangerous explosive. Because existing benchmarks often overlook these cross-modality interactions, they fail to expose how benign inputs can synthesize harmful outcomes.
The Alignment Tax of Visual Instruction Tuning
Intuitively, teaching a model to see should not make it forget how to behave. Yet, research shows that the process of adapting an LLM into a VLM inherently compromises its prior safety training. Georgios Pantazopoulos et al. describe this phenomenon as "learning to see but forgetting to follow."
When a model undergoes visual instruction tuning, the optimization process shifts its internal representations to accommodate multimodal tokens. This shift induces a "forgetting effect" on the model's text-based safety guardrails, leaving the resulting VLM significantly more susceptible to jailbreak attacks than its text-only predecessor. Consequently, safety evaluations that assume a VLM inherits its base LLM's robustness are fundamentally flawed; new benchmarks must explicitly test for both visual and textual adversarial vectors simultaneously.
Superficial Associations vs. Deep Visual Grounding
Another reason for the evaluation gap is that standard performance metrics mistake superficial correlation for genuine understanding. In safety-critical domains like autonomous driving, a VLM might generate a fluent, correct-sounding explanation of a driving scene. However, Shaoyuan Xie et al. found that these models often rely on general textual knowledge rather than actual visual grounding, especially under degraded conditions. Standard NLP evaluation metrics are easily fooled by this linguistic fluency, masking the model's inability to reason about rare corner cases or accidents—a gap highlighted by Hao Ye et al. and Ana-Maria Marcu et al..
This reliance on superficial shortcuts makes VLMs highly brittle. Carlos Hinojosa et al. demonstrated that simple visual or textual semantic cues can easily steer a VLM's safety judgments, proving that the models rely heavily on learned associations rather than robust situational awareness. Similarly, Haoqin Tu et al. and the authors of How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMs observed that while VLMs generalize well to out-of-distribution (OOD) images, they struggle significantly with OOD text and are highly vulnerable to adversarial attacks targeting their vision encoders.
Why Existing Benchmarks Fail to Close the Gap
If the safety gap is so severe, why haven't standard evaluation suites caught it? The literature points to three systemic flaws in current benchmarks:
- Low-Quality and Generic Queries: Many existing safety datasets rely on generic, low-quality harmful queries that fail to simulate realistic, sophisticated threats. As argued by Baolin Zheng et al. and Wenbo Zhou et al., these simplistic evaluations mask true vulnerabilities, creating a false sense of security.
- Static and Fragmented Frameworks: The landscape of adversarial threats changes rapidly. Lu Zhou et al. note that while many benchmarks exist, they lack a unified evaluation system, leading to fragmented assessments. Furthermore, static benchmarks quickly become obsolete or suffer from data leakage. As highlighted by Mukai Li et al., there is a severe lack of systematic frameworks for stress-testing VLMs across diverse domains like privacy, fairness, and adversarial robustness. To combat this, Xiangyang Zhu et al. emphasize the need for evolvable, multi-agent pipelines that dynamically update safety benchmarks to maintain discriminative power.
- Lack of Modern, Complex Threat Vectors: Modern risks involve complex, AI-generated content and regulatory compliance. For instance, Bhavik Chandna et al. point out that benchmarks lack AI-generated extremist content and diverse image generation models, while Shaina Raza et al. highlight the absence of frameworks aligned with AI governance principles for detecting multimodal disinformation. Even text-to-image models lack standardized, open safety benchmarks to evaluate automated red-teaming prompts, as noted by the authors of AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Prompts.
Bridging the safety evaluation gap requires moving away from isolated, static, and text-centric metrics. True multimodal safety evaluation demands holistic frameworks that explicitly test cross-modal interactions, out-of-distribution robustness, and dynamic adversarial red-teaming. Only by evaluating VLMs through the lens of emergent multimodal vulnerabilities can we build systems that remain reliably safe in the real world.
Key Foundational Work
Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks (2024) — contributed 3 concept elements to this direction.
SafeDriveRAG: Towards Safe Autonomous Driving with Knowledge Graph-based Retrieval-Augmented Generation (2025) — contributed 2 concept elements to this direction.
How Many Are in This Image A Safety Evaluation Benchmark for Vision LLMs (2024) — contributed 2 concept elements to this direction.