Efficient LLM-Jailbreaking by Introducing Visual Modality

Zhenxing Niu, Yuyao Sun, Haodong Ren, Haoxuan Ji, Quan Wang, Xiaoke Ma, Gang Hua, Rong Jin

Abstract

This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreaks that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) through the incorporation of a visual module into the target LLM. Subsequently, we conduct an efficient MLLM-jailbreak to generate jailbreaking embeddings $JB_{emb}$. Finally, we convert the $JB_{emb}$ into text space to facilitate the jailbreaking of the target LLM. Compared to direct LLM-jailbreaking, our approach is more efficient, as MLLMs are more vulnerable to jailbreaking than pure LLM. Additionally, to improve the attack success rate (ASR) of jailbreaking, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class jailbreaking capabilities.

Introduction

Jailbreaking an LLM—forcing it to bypass safety filters to answer harmful queries—is traditionally a "discrete optimization" problem. Attackers typically search for a specific string of text (a suffix) that triggers the model. Because text consists of individual tokens, searching for the right combination is computationally expensive and slow.

Researchers have discovered a "backdoor" for jailbreaking LLMs by temporarily turning them into Multimodal LLMs (MLLMs). Multimodal models are known to be more vulnerable because images exist in a continuous pixel space where mathematical gradients can easily guide an attack. This research leverages the "softness" of multimodal vulnerabilities to break pure-text LLMs more efficiently, achieving state-of-the-art success rates while requiring only 3% of the computation time of previous methods.

Methodology

The proposed indirect jailbreaking scheme consists of four main stages designed to translate continuous multimodal optimizations into discrete textual attacks.

Overview of the proposed jailbreaking pipeline, showing the transition from multimodal embeddings to textual suffixes.
  1. Step 0: Finding a Starting Point ($JB_{init}$): Instead of starting with a random image, a CLIP-based semantic matching scheme finds an image related to the harmful query (e.g., a photo of a firearm for a query about weapons).
  2. Step 1: MLLM Construction & Jailbreak: The target LLM is augmented with a visual module (like a CLIP encoder) to create a temporary MLLM. A fast jailbreak is performed on this multimodal version using Projected Gradient Descent (PGD) to find a "jailbreaking embedding" ($JB_{emb}$).
  3. Step 2: De-embedding and De-tokenization: The continuous $JB_{emb}$ is mapped back to the nearest discrete text tokens in the LLM’s vocabulary, creating a textual jailbreak suffix ($JB_{txt}$).
  4. Step 3: The Final Attack: This $JB_{txt}$ is appended to harmful queries and sent to the original, text-only target LLM.

The core innovation is using the visual module as a regularizer. Standard text-embedding attacks often produce "gibberish" that doesn't map well to real tokens. By optimizing through a visual encoder trained on natural images, the resulting embeddings stay closer to the distribution of real language, making the final text suffix much more effective.

Experimental Evaluation

The study utilized benchmarks including AdvBench, HarmBench, and a custom AdvBench-M dataset (augmented with semantically relevant images). The results demonstrate a significant leap in performance over existing baselines like GCG, AutoP, and AdvA.

Comparison of Attack Success Rate (ASR) and running time on LLaMA2 across different harmful categories.

Key Results

  • Massive Speedup: The proposed method is roughly 30 times faster than GCG. It takes approximately 22 minutes (0.37h) to generate a suffix that previously required over 11 hours.
  • Higher Success Rates: On LLaMA2, the method achieved an 84.7% average ASR on test sets, significantly outperforming GCG (74.7%) and AdvA (47.3%).
  • Black-Box Transferability: Suffixes generated on open-source surrogate models successfully jailbroke proprietary models, including Mistral-v0.3 (77.1% ASR) and DeepSeek-R1 (70.5% ASR).

Discussion and Conclusion

The research confirms a "Multimodal Vulnerability Gap," where MLLMs are significantly easier to jailbreak than pure LLMs. By "borrowing" this vulnerability, attackers can bypass the difficult discrete search space of text. Furthermore, the visual module acts as a "language anchor," constraining the attack to produce embeddings that align with natural language, thereby reducing projection errors during de-tokenization.

While highly effective, some proprietary models like Grok-3 and DouBao-1.5 showed higher resistance, suggesting more robust internal defensive mechanisms. Additionally, categories like "Hate Speech" and "Self-Harm" remain difficult to jailbreak, likely due to intensive safety alignment. Future safety research should focus on "cross-class" alignment to ensure that hardening one type of harmful query also protects against related categories.

Summary updated

Abstract

This paper focuses on jailbreaking attacks against large language models (LLMs), eliciting them to generate objectionable content in response to harmful user queries. Unlike previous LLM-jailbreaks that directly orient to LLMs, our approach begins by constructing a multimodal large language model (MLLM) through the incorporation of a visual module into the target LLM. Subsequently, we conduct an efficient MLLM-jailbreak to generate jailbreaking embeddings embJS. Finally, we convert the embJS into text space to facilitate the jailbreaking of the target LLM. Compared to direct LLM-jailbreaking, our approach is more efficient, as MLLMs are more vulnerable to jailbreaking than pure LLM. Additionally, to improve the attack success rate (ASR) of jailbreaking, we propose an image-text semantic matching scheme to identify a suitable initial input. Extensive experiments demonstrate that our approach surpasses current state-of-the-art methods in terms of both efficiency and effectiveness. Moreover, our approach exhibits superior cross-class jailbreaking capabilities.
BibTeX
@inproceedings{niu2026efficient,
  author = {Zhenxing Niu and Yuyao Sun and Haodong Ren and Haoxuan Ji and Quan Wang and Xiaoke Ma and Gang Hua and Rong Jin},
  title = {{Efficient LLM-Jailbreaking by Introducing Visual Modality}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i42.40863},
}
Figure