Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

Zhaoxin Wang, Handing Wang, Cong Tian, Yaochu JinarXiv 2025 · DOI · arXiv

Abstract

Multimodal Large Language Models (MLLMs) integrate visual and textual processing, expanding the potential attack surface for malicious actors. While safety alignment has improved, this paper identifies a critical vulnerability: Implicit Jailbreak Attacks (IJA). By concealing harmful instructions within images using steganography, attackers can bypass standard safety filters and force models to decode and execute malicious payloads. IJA achieves over 90% success rates on leading models like GPT-4o and Gemini 1.5 Pro, highlighting a significant gap in current multimodal safety defenses.

Introduction

The evolution of Large Language Models into Multimodal variants (MLLMs) has introduced new challenges for safety alignment. Attackers often exploit less-aligned modalities, such as images, to inject harmful instructions. Previous methods, including typographic attacks (text on images) and adversarial noise, are increasingly detectable by modern OCR and image safety classifiers. This research introduces the Implicit Jailbreak Attack (IJA), a framework designed to deliver malicious payloads without appearing suspicious to human reviewers or automated filters.

Methodology

The IJA framework operates through a sophisticated three-stage process designed to exploit the cross-modal reasoning capabilities of MLLMs:

The three-step Implicit Jailbreak Attack (IJA) framework.
  1. Textual Malicious Transfer: An adversarial suffix is generated using a surrogate model and the Greedy Coordinate Gradient (GCG) algorithm. This suffix is designed to steer the target model toward compliance with harmful requests.
  2. Visual Malicious Embedding: The harmful instruction and its adversarial suffix are converted into binary data and embedded into an image using Least Significant Bit (LSB) steganography. This ensures the image remains visually benign to both humans and standard safety filters.
  3. Attack Template Optimization: The steganographic image is paired with an innocent-looking extraction prompt. If the initial attempt fails, an LLM judge iteratively refines the prompt to improve decoding accuracy (e.g., by specifying exact bit lengths or formatting instructions).

Experimental Setup

The effectiveness of IJA was tested across three comprehensive safety benchmarks:

  • SafeBench: 500 harmful questions covering 10 prohibited topics.
  • MM-SafetyBench: 1,180 malicious samples across 13 categories.
  • HADES: 750 malicious instructions specifically designed to test typographic jailbreak attacks.

Key Results

The proposed attack demonstrates remarkable efficacy and stealth. On the HADES dataset, IJA achieved an Attack Success Rate (ASR) of 91.20% on GPT-4o and 95.07% on Gemini 1.5 Pro.

Comparison of Attack Success Rate and Average Queries across different jailbreak methods.

Beyond high success rates, the attack is highly efficient, requiring an average of only ~2.1 queries to succeed. Most importantly, it achieved a 97.69% bypass rate against standard safety filters, as both the text prompt and the image appear harmless in isolation. Interestingly, the open-source Qwen2.5-VL showed higher resilience (65.87% ASR), primarily due to its lower proficiency in following complex cross-modal decoding instructions rather than superior safety alignment.

Discussion and Conclusion

The vulnerability of MLLMs to IJA stems from their core strength: the ability to reason across modalities. By tasking the model with decoding hidden data, attackers force the model to reconstruct the harmful payload internally, effectively bypassing input-level defenses. While currently limited to lossless image formats, the success of IJA underscores the need for more robust, cross-modal safety checks that account for implicit information concealment. Future research will explore extending these vulnerabilities to audio and video modalities.

Summary updated

Abstract

Multimodal large language models (MLLMs) enable powerful cross-modal reasoning capabilities. However, the expanded input space introduces new attack surfaces. Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision. As MLLMs increasingly incorporate cross-modal consistency and alignment mechanisms, such explicit attacks become easier to detect and block. In this work, we propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via least significant bit steganography and couples them with seemingly benign, image-related textual prompts. To further enhance attack effectiveness across diverse MLLMs, we incorporate adversarial suffixes generated by a surrogate model and introduce a template optimization module that iteratively refines both the prompt and embedding based on model feedback. On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.
BibTeX
@misc{wang2025implicit,
  author = {Zhaoxin Wang and Handing Wang and Cong Tian and Yaochu Jin},
  title = {{Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models}},
  year = {2025},
  eprint = {2505.16446},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.16446},
}
Figure