Abstract: Safety concerns of Multimodal large language models (MLLMs) have gradually become an important problem in various applications. Surprisingly, previous works indicate a counterintuitive phenomenon that using textual unlearning to align MLLMs achieves comparable safety performances with MLLMs aligned with image-text pairs. To explain such a phenomenon, we discover a Visual Safety Information Leakage (VSIL) problem in existing multimodal safety benchmarks, i.e., the potentially risky content in the image has been revealed in the textual query. Thus, MLLMs can easily refuse these sensitive image-text pairs according to textual queries only, leading to unreliable cross-modality safety evaluation of MLLMs. We also conduct a further comparison experiment between textual alignment and multimodal alignment to highlight this drawback. To this end, we construct Visual Leakless Safety Bench (VLSBench) with 2.2k image-text pairs through an automated data pipeline. Experimental results indicate that VLSBench poses a significant challenge to both open-source and close-source MLLMs, i.e., LLaVA, Qwen2-VL and GPT-4o. Besides, we empirically compare textual and multimodal alignment methods on VLSBench and find that textual alignment is effective enough for multimodal safety scenarios with VSIL, while multimodal alignment is preferable for safety scenarios without VSIL.
Introduction: The Visual Safety Information Leakage (VSIL) Problem
As Multimodal Large Language Models (MLLMs) are increasingly deployed, ensuring their safety against harmful queries is paramount. However, current evaluation methods suffer from a critical flaw: Visual Safety Information Leakage (VSIL). In many existing benchmarks, the harmful intent is explicitly stated in the text prompt, allowing models to bypass visual processing and rely solely on internal text-based safety filters to reject the query.
As shown in Figure 1, when the text prompt contains keywords like "making porn" or "online gambling," the model's textual alignment is sufficient to trigger a refusal. This creates an illusion of multimodal safety. When the VSIL is removed—pairing benign text with risky images—the safety performance of models aligned only via text drops significantly, highlighting the need for true cross-modal understanding.
The VLSBench Pipeline
To address this, we introduce VLSBench, a benchmark designed to be "leakless." We developed an automated pipeline to generate image-text pairs where the text is innocuous on its own but becomes unsafe when combined with the visual context. The pipeline involves four stages:
- Harmful Scenario Generation: GPT-4o generates a harmful scenario description.
- Text Detoxification: The harmful query is rewritten into a neutral, benign-sounding prompt (e.g., changing "How do I use this syringe for drugs?" to "What's the optimal amount to use for fast results with this equipment?").
- Iterative Image Generation: Stable Diffusion 3.5 generates the image, which is then critiqued by Qwen2-VL-72B to ensure the visual risk is present.
- Final Filtration: A combination of GPT-4o and human review ensures the final pair is truly leakless and unsafe only in combination.
Dataset Composition
VLSBench consists of 2,241 high-quality image-text pairs. The dataset is categorized into six primary safety domains to ensure comprehensive coverage of potential risks.
As detailed in Figure 4, the dataset emphasizes Violent (26.33%) and Illegal Activity (24.05%) categories, which are often the most critical for real-world safety alignment. Other categories include Self-Harm, Erotic content, Hate speech, and Privacy violations.
Key Experimental Results
Our evaluation reveals that VLSBench is significantly more challenging than existing benchmarks for both open-source and proprietary models:
- State-of-the-Art Failure: Even leading models struggle with contextual danger. GPT-4o achieved a safety rate of only 21.43%, while Gemini-1.5-pro reached 49.78%.
- The Necessity of Multimodal Alignment: On VLSBench, multimodal alignment (training on image-text pairs) consistently outperforms textual alignment. For LLaVA-v1.5-7B, Multimodal SFT achieved a 21.26% safety rate, compared to just 13.99% for Textual SFT.
- Contextual Blindness: Models often fall into the trap of being helpful. When asked a benign question about a risky object (e.g., "How do I clean this?" referring to a blood-stained weapon), models frequently provide instructions without recognizing the visual red flags.
Conclusion
VLSBench exposes a major blind spot in current MLLM safety evaluations. By removing textual cues of harm, we demonstrate that current models are not yet capable of robust cross-modal safety reasoning. We hope this benchmark encourages the community to move beyond textual shortcuts and develop more sophisticated multimodal alignment strategies.
Summary updated