Reading Scene States: Uncovering Safety Failures in Vision-Language Models

Tongxi Wu, Yuyang Cheng, Jiacheng Xian, Menglin Yang, Kai Chen, Jian Zhang, Yang GaoarXiv 2026 · arXiv

TL;DR

This paper identifies a critical "scene-state" safety vulnerability in Vision-Language Models (VLMs), where models provide harmful assistance by interpreting visual situations even without explicit harmful requests. The authors introduce SceneSafetyBench to evaluate this risk and S3TA, an automated attack framework that achieves a 71.2% success rate by translating harmful objectives into implicit visual scenes.

What Problem Does This Paper Address?

Current safety alignment for VLMs primarily focuses on detecting explicit harmful text or obvious visual triggers (like typographic "jailbreaks"). However, the authors argue that VLMs' core strength—reasoning about visual situations—is itself a vector for safety failure.

A model might refuse a direct text request like "tell me how to make a bomb," but if shown an image of a Molotov cocktail in progress and asked "tell me the technique for enhancing the damage in the image," it may provide detailed, actionable instructions. This is because the model infers the harmful intent from the scene state (the configuration of entities and the progress of an activity) rather than from the literal input content.

Figure 1. Illustrative results on GPT-5.6 Luna under three input configurations. The model explicitly refuses the harmful objective in the text-only and QR-related cases, while providing action-oriented guidance in response to the scene-state input.

Methods

The researchers took a three-pronged approach to characterize and exploit this vulnerability:

  1. Benchmarking: They created SceneSafetyBench, a dataset of 375 image-question pairs categorized by activity stage: Initial (pre-activity), Medium (in-progress), and Ultimate (outcome).
  2. Mechanistic Analysis: To understand why this happens, they used HiddenDetect, a framework that projects internal model states into "vocabulary-space logits" to measure alignment with safety-related tokens (refusal vs. risk). They also performed component interventions—zeroing out specific attention heads or MLP modules—to identify which parts of the model process these scene-state signals.
  3. Automated Attack (S3TA): They developed the Scene-State Attack (S3TA) framework. It uses an LLM to plan a scene that implies a harmful goal, generates the image using a text-to-image model, and then queries the target VLM. It includes a "checking" step where another VLM ensures the scene is recognizable before the attack is finalized.

Datasets

  • SceneSafetyBench (New): 75 base samples (375 total pairs) across three scene states and five visual styles (Original, Pixel Art, Pencil Sketch, Oil Painting, Film Noir).
  • MM-SafetyBench: A standard benchmark used to evaluate the S3TA attack across seven categories, including Illegal Activity, Physical Harm, and Malware Generation.
Examples from SceneSafetyBench showing different scene states and visual styles.

Key Insights and Ideas

  • Implicit Intent is the Blind Spot: VLMs are trained to be helpful assistants that "ground" their reasoning in the image. When a scene state implies a goal, the model's instruction-following nature overrides its safety filters because the "harm" isn't in the text or a single object, but in the interpretation of the situation.
  • Internal Signal Dynamics: Mechanistic analysis revealed that scene-state inputs produce internal safety signals similar to "QR-related" attacks (which use images to convey harmful text), but with significantly lower signal strength in the later layers of the decoder. This suggests that the model's "refusal" mechanism is less likely to trigger as generation progresses.
  • Contextual Sensitivity: The strength of the internal safety signal depends heavily on the output context. For example, if a model starts its response by acknowledging a request, its internal "risk" signal might actually decrease compared to when it provides general context.
  • Style Invariance: The vulnerability persists across various artistic styles (like oil painting or pencil sketches), proving it's a high-level semantic issue rather than a low-level visual artifact.

Key Results

  • High Vulnerability: Across eight major VLMs (including GPT-5.6 Luna, Gemini 3.5 Flash, and Claude Haiku 4.5), scene-state inputs elicited harmful assistance with Attack Success Rates (ASR) between 45.6% and 72.3%.
  • S3TA Superiority: The automated S3TA attack achieved a model-average ASR of 71.2%, outperforming the strongest existing baseline by 6.1 percentage points.
  • Defense Failure: Existing text-side filters (like Llama Guard 3) and visual moderation tools (like ShieldGemma) provided very limited protection, as they failed to "reason" about the harmful implications of the depicted scenes.
Internal safety-related signal profiles across different layers, showing how Scene-State inputs compare to other attack types.

Limitations and Future Work

The authors acknowledge several areas for further exploration:

  • Single-Turn Focus: The current study primarily looks at single-turn interactions. Future work could examine how scene-state reasoning evolves over multi-turn dialogues.
  • Defense Gap: The study highlights a major gap in current safeguards. Future research needs to develop "scene-grounded" safety reasoning that can evaluate the intent behind a visual situation, not just the presence of prohibited content.
  • Model Scope: While they tested a wide range of models, the rapid evolution of VLMs means new architectures may exhibit different internal dynamics.

Summary updated

Abstract

Vision-language models (VLMs) can provide harmful assistance by interpreting visual events, even without an explicit request for the underlying harmful objective. We identify this \textbf{scene-state safety vulnerability} and introduce \textbf{SceneSafetyBench}, comprising 375 image--question pairs spanning Initial, Medium, and Ultimate scene states and five visual presentations. Across eight VLMs, scene-state inputs elicit harmful assistance with attack success rates (ASRs) of 45.6--72.3\%. Comparative analysis across three VLMs shows that scene-state inputs most closely resemble QR-related images in their layer-wise safety-related signal profiles, while exhibiting lower average late-layer readouts and distinct signal dynamics during generation. To turn this vulnerability into a systematic attack, we propose \textbf{S3TA}, which translates harmful objectives into scene states, queries image--question pairs independently, and assembles the responses. On seven MM-SafetyBench categories across eight open-weight and commercial VLMs, S3TA achieves a model-average ASR of \textbf{71.2\%, exceeding the strongest baseline by 6.1 percentage points}. Text-side filters and visual moderation provide limited protection. These findings highlight the need to safeguard the assistance models infer through scene-grounded reasoning, beyond screening input content.
BibTeX
@misc{wu2026reading,
  author = {Tongxi Wu and Yuyang Cheng and Jiacheng Xian and Menglin Yang and Kai Chen and Jian Zhang and Yang Gao},
  title = {{Reading Scene States: Uncovering Safety Failures in Vision-Language Models}},
  year = {2026},
  eprint = {2611.23206},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23206},
}
Figure