Abstract
With the rapid advancement of multimodal large language models (MLLMs), concerns regarding their security have increasingly captured the attention of both academia and industry. Although MLLMs are vulnerable to jailbreak attacks, designing effective multimodal jailbreak attacks poses unique challenges, especially given the distinct protective measures implemented across various modalities in commercial models. Previous works concentrate risks into a single modality, resulting in limited jailbreak performance. In this paper, we propose a heuristic-induced multimodal risk distribution jailbreak attack method, called HIMRD, which consists of two elements: multimodal risk distribution strategy and heuristic-induced search strategy. The multimodal risk distribution strategy is used to segment harmful instructions across multiple modalities to effectively circumvent MLLMs' security protection. The heuristic-induced search strategy identifies two types of prompts: the understanding-enhancing prompt, which helps the MLLM reconstruct the malicious prompt, and the inducing prompt, which increases the likelihood of affirmative outputs over refusals, enabling a successful jailbreak attack. Extensive experiments demonstrate that this approach effectively uncovers vulnerabilities in MLLMs, achieving an average attack success rate of 90% across seven popular open-source MLLMs and an average attack success rate of around 68% in three popular closed-source MLLMs.
Introduction
As Multimodal Large Language Models (MLLMs) like GPT-4o and Gemini become more integrated into complex applications, their safety alignment—the "guardrails" designed to prevent the generation of harmful content—is under increasing scrutiny. Current jailbreak attacks typically focus on a single modality, either using "adversarial suffixes" in text or hiding malicious instructions inside images via typography attacks.
However, these concentrated risks are often easily detected by modality-specific safety filters. This research identifies a critical gap: MLLM safety mechanisms struggle when harmful intent is fragmented. By distributing pieces of a malicious prompt across both text and image modalities, an attacker can ensure that neither part looks harmful in isolation. The model only "reconstructs" the harmful intent during the inference process, effectively bypassing input filters.
Methodology: The HIMRD Attack
The authors propose the Heuristic-Induced Multimodal Risk Distribution (HIMRD) attack, which consists of two main components designed to exploit the modality-blind nature of current safety filters.
1. Multimodal Risk Distribution Strategy
A malicious prompt (e.g., "make illegal drugs") is decomposed into two harmless-looking segments. One segment is embedded into a text template (e.g., "make illegal ( )"), while the other is rendered into an image using typography (e.g., the word "drugs"). To further distract safety filters and provide semantic context, an auxiliary LLM is used to generate a contextually relevant background image, such as a person in a laboratory setting.
2. Heuristic-Induced Search Strategy
Since the model might not immediately connect the fragmented inputs, the authors employ an iterative, black-box search to refine the text prompt. This search optimizes two specific types of prompts:
- Understanding-enhancing prompt ($p_u$): Guides the model to correctly reconstruct the full malicious intent from the split parts.
- Inducing prompt ($p_i$): Encourages the model to adopt an affirmative tone (e.g., "Sure, I can help with that") and ignore its internal safety constraints.
Experimental Evaluation
The researchers evaluated HIMRD using benchmarks such as SafeBench and SORRY-Bench, covering hundreds of harmful questions across dozens of categories. The results demonstrate significant vulnerabilities in both open and closed-source models:
- Open-Source Dominance: HIMRD achieved an average 90.86% Attack Success Rate (ASR) across seven open-source models, including LLaVA-V1.6 and DeepSeek-VL.
- Closed-Source Vulnerability: The attack is remarkably effective against "frontier" models, achieving a 44.29% ASR on GPT-4o, 64.29% on Gemini-1.5-Pro, and a massive 95.71% on Qwen-VL-Max.
- Efficiency and Resilience: Compared to other state-of-the-art attacks, HIMRD is significantly faster, generating successful samples in roughly 32 seconds. It also remained effective against common countermeasures like image denoising or perplexity-based text filters.
Discussion and Conclusion
The success of HIMRD highlights a "Modality-Blind" safety gap where MLLM safety filters often act as independent gatekeepers. By ensuring neither gatekeeper sees a "red flag," the attack slips through. Visualizations of the model's internal representations show that while initial inputs appear harmless, the model's internal state converges toward harmful representations as the heuristic search progresses.
While highly effective, the attack shows some sensitivity to content categories; for instance, it performed slightly worse on pornography, suggesting more robust alignment for explicit visual content. Ultimately, the paper concludes that current MLLM defenses are too modality-centric. Future work must focus on "cross-modal" safety alignment that can detect distributed risks before they are reconstructed by the model's reasoning engine.
Summary updated