Abstract
Large Vision-Language Models (LVLMs) have demonstrated remarkable capabilities in multimodal understanding, yet their safety alignment remains predominantly focused on explicit unimodal threats. This paper identifies a critical vulnerability termed Implicit Reasoning Safety (IRS), where individually benign images and text prompts combine to form harmful semantic interpretations. To address this, we introduce the SSUI (Safe Semantics, Unsafe Interpretations) dataset, a novel benchmark designed to evaluate and mitigate these cross-modal risks. Our findings reveal that even state-of-the-art models like GPT-4o and Gemini-1.5 are susceptible to these implicit threats. However, we demonstrate that applying In-Context Learning (ICL) using the SSUI dataset significantly enhances model safety and reasoning transparency without requiring extensive retraining.
Introduction
Current safety protocols for Large Vision-Language Models (LVLMs) are largely designed to detect and refuse explicitly harmful content, such as violent imagery or toxic text. While effective against unimodal attacks, these safeguards often fail to recognize threats that emerge only through the interaction of safe components. For instance, an image of a high-speed railway track is inherently benign, as is a text prompt encouraging someone to "take a unique photo." However, when paired, the combination implicitly encourages life-threatening behavior.
This failure of Implicit Reasoning Safety (IRS) represents a significant blind spot in AI alignment. Because the individual inputs do not trigger standard safety filters, models may inadvertently generate harmful content or endorse dangerous activities. This paper formalizes the concept of IRS and provides a framework for benchmarking and improving the cross-modal reasoning necessary to navigate these nuanced scenarios.
The SSUI Dataset and Construction Pipeline
To systematically study IRS, we developed the Safe Semantics, Unsafe Interpretations (SSUI) dataset. The dataset covers nine hazard classes, including Dangerous Behavior, Moral Violations, and Political Controversies. To ensure high quality and interpretable reasoning, we employed a 5-stage AI-assisted pipeline:
- Initial Query Formulation: Utilizing GPT-4o to analyze benign images and hypothesize innocuous text prompts that, when combined with the image, create an unsafe implication.
- Explainable Reasoning Generation: GPT-4o generates a step-by-step "Chain of Thought" (CoT) explaining the latent danger. This ensures the dataset provides not just a safety label, but a reasoning path.
- Reflective Query Refinement: Queries are iteratively refined to ensure the model must actually process the image to understand the threat, preventing the model from relying solely on text-based heuristics.
- Text-Only Safety Validation: We use a text-only model (DeepSeek-R1) to verify that the prompt alone is safe. If the text triggers a safety warning without the image, the pair is discarded to maintain the strictly multimodal nature of the benchmark.
- Manual Curation: Human experts review the final pairs to ensure they meet the difficulty and quality standards required for a robust benchmark.
Images for SSUI were sourced from established public datasets, including Open Images v7, COCO, and EgoShots, ensuring a diverse range of real-world contexts.
Mitigating Risks via In-Context Learning
Rather than focusing on computationally expensive fine-tuning, we explore the efficacy of In-Context Learning (ICL) as a mitigation strategy. By providing LVLMs with a few-shot prompt containing examples from the SSUI dataset—complete with their explainable reasoning paths—we aim to "prime" the models to perform deeper cross-modal analysis at inference time.
Experimental Results
We evaluated several leading LVLMs using two primary metrics: Safety Rate (S), the percentage of safe refusals, and Effectiveness Rate (E), the percentage of responses that remain helpful and relevant rather than defaulting to over-refusal.
The results indicate a significant vulnerability in zero-shot performance across all models, followed by a dramatic improvement when ICL is applied:
- GPT-4o: Zero-shot safety was measured at 58.88%. With ICL, this rose to 80.90%, while effectiveness simultaneously improved from 92.06% to 97.81%.
- Gemini-1.5: Safety increased from 55.10% to 75.52%.
- Qwen2.5-VL: This open-source model saw the most substantial gain, jumping from 50.90% to 70.72% safety.
On average, the inclusion of SSUI examples in the prompt improved the combined safety and effectiveness scores by 15 to 20 percentage points. This suggests that LVLMs possess the latent reasoning capacity to identify implicit threats but require specific guidance to activate these pathways.
Conclusion and Future Work
The SSUI benchmark highlights a critical gap in the current multimodal safety landscape. While In-Context Learning provides a powerful and immediate mitigation strategy, it is not a total solution. The reasoning processes of LVLMs remain largely opaque, and their susceptibility to implicit threats suggests that safety must be integrated more deeply into the training and alignment phases. Future research should focus on developing architectural or algorithmic improvements that allow LVLMs to inherently perform the cross-modal reasoning required for safety, reducing the reliance on prompt-based interventions.
Summary updated