Abstract
Large Vision-Language Models (LVLMs) enhance the capabilities of Large Language Models by integrating visual inputs, thereby enabling advanced multimodal reasoning across diverse applications. However, these enhanced reasoning capabilities introduce new security risks, particularly to jailbreaking attacks that bypass built-in safety mechanisms to elicit harmful or unauthorized outputs. In this work, we propose TVChain, a novel black-box jailbreaking framework that explicitly intervenes in both the visual and textual reasoning processes of LVLMs. TVChain decomposes malicious prompts into a sequence of semantically meaningful sub-images that represent relevant objects and behaviors, thereby circumventing direct exposure of illicit content. In parallel, a carefully designed chain-of-thought (CoT) textual prompt is employed to steer the model's reasoning toward reconstructing the intended activity in a covert yet effective manner. Extensive evaluations on eleven LVLMs across two benchmark datasets and three state-of-the-art defenses validate the effectiveness and robustness of TVChain.
Introduction
As Large Vision-Language Models (LVLMs) become more sophisticated, they integrate visual and textual reasoning to solve complex tasks. However, this multimodal integration opens new security backdoors. Current "jailbreaking" attempts—which try to trick a model into generating prohibited content (e.g., instructions for hacking or violence)—typically rely on adversarial images (adding noise) or typographic attacks (embedding text).
These methods suffer from a "smoking gun" problem: they concentrate the harmful intent into a single image, making them easy for safety filters like Llama Guard to detect. Furthermore, many existing attacks require "white-box" access to model weights, which is impractical for commercial APIs like GPT-4o.
Methodology: The TVChain Framework
TVChain introduces a "divide and conquer" strategy called Textual-Visual Prompt Chains. Instead of presenting a single malicious image, it uses a sequence of neutral components to bypass safety heuristics.
Visual Prompt Chain
The framework first extracts core objects and behaviors from a malicious query using an LLM. For example, a query about "hacking a bank account" is decomposed into elements like "a hacker," "a bank server," and "a person typing." A text-to-image model then generates individual sub-images for each element, which are arranged into a composite grid. Individually, these images appear benign; the harmful intent only emerges when the LVLM views them as a sequence.
Textual Prompt Chain
To ensure the LVLM connects these visual dots, TVChain employs a 5-step Chain-of-Thought (CoT) scaffold:
- Observe: Examine each panel in the grid.
- Identify: Determine the actors and actions involved.
- Link: Connect the actions across panels.
- Infer Goal: Deduce the overall purpose of the sequence.
- Output: Provide the final detailed guide based on the inferred goal.
By following this disciplined reasoning path, the model effectively "talks itself" into the jailbreak, bypassing the abrupt semantic leaps that usually trigger safety filters.
Key Results and Evaluation
TVChain was evaluated against 11 models (7 open-source, 4 commercial) using the HADES and MM-Safety benchmarks. The results demonstrate that the very reasoning capabilities that make LVLMs powerful are the exact mechanisms TVChain exploits.
- High Attack Success Rates (ASR): On the HADES dataset, TVChain achieved an ASR of over 90% on several open-source models. Commercial models were also vulnerable, with GPT-4o seeing an ASR of 74.13% and Gemini 2.0 FT reaching 82.27%.
- Evasion of Defenses: TVChain proved significantly harder to detect than traditional attacks. Against Llama Guard 3, TVChain's detection rate was markedly lower than that of the HADES baseline across all categories.
Conclusion
TVChain demonstrates that distributed harm is far stealthier than concentrated harm. By spreading malicious intent across a grid of sub-images and leveraging the model's own reasoning to reconstruct that intent, the attack stays below the detection threshold of state-of-the-art defenses. Future safety filters must evolve to perform "compositional" safety checks that look for harmful patterns across sequences of images rather than analyzing them in isolation.
Summary updated