CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attacks

Xu Zhang, Hao Li, Zhichao LuarXiv 2025 · DOI · arXiv

Introduction: The Modality Gap in MLLM Safety

Current safety guardrails for Multimodal Large Language Models (MLLMs) are proficient at identifying explicit attacks—queries where the harmful intent is obvious in either the text or the image alone. However, a significant security hole exists in the form of joint-modal implicit attacks. In these scenarios, the text and image are individually benign, but their combination reveals a malicious intent. For example, a request for "guidance on climbing this wall" paired with an image of a prison wall bypasses standard filters because the "malice" only emerges through cross-modal reasoning.

Figure 1: Comparison of explicit vs. implicit malicious attacks and the resulting Attack Success Rate (ASR) across models.

Methodology: ImpForge and CrossGuard

To address this, researchers developed a two-stage framework: ImpForge for data generation and CrossGuard for defense.

1. ImpForge (The Red-Teamer)

The primary bottleneck in defending against implicit attacks is the lack of high-quality training data. ImpForge is an automated pipeline that uses Reinforcement Learning (PPO) to transform known malicious prompts into subtle, implicit versions. It selects benign images related to malicious keywords and rewrites the text to be "individually safe" but "jointly malicious." The RL agent is optimized via three rewards:

  • Safety Reward: Ensures the text alone does not trigger standard guardrails.
  • Semantic Reward: Maintains the original malicious intent in the multimodal combination.
  • Overlap Reward: Prevents the text from simply describing the image, ensuring the connection remains implicit.

2. CrossGuard (The Defender)

CrossGuard is a LLaVA-1.5-7B model fine-tuned using LoRA on the diverse dataset generated by ImpForge. It serves as a front-end safety filter, classifying multimodal queries as safe or unsafe before they reach the primary MLLM.

Evaluation and Results

CrossGuard was evaluated against several benchmarks, including SIUO (implicit attacks) and VLGuard (explicit attacks). Across five different safety benchmarks, CrossGuard maintained an average Attack Success Rate (ASR) of 2.79%, outperforming all other open-source and proprietary baselines.

  • State-of-the-Art Defense: On the SIUO benchmark, CrossGuard achieved a remarkably low ASR of 5.39%. In contrast, proprietary models like GPT-4o (48.92% ASR) and specialized guardrails like Llama-Guard3-Vision (89.82% ASR) struggled significantly.
  • The Safety-Utility Pareto Frontier: A common pitfall for guardrails is "over-defense," where the model becomes too restrictive and blocks safe queries. CrossGuard achieves an optimal balance, maintaining high utility on safe samples (evaluated via MMBench) while providing robust protection against unsafe ones.
Figure 3: The trade-off between utility on safe samples and safety (1-ASR) on unsafe samples, showing CrossGuard's superior performance relative to baselines.

Conclusion

The "modality gap" represents a critical vulnerability in modern MLLMs. By leveraging RL-based red-teaming to bridge this gap, CrossGuard demonstrates that it is possible to defend against sophisticated implicit threats without compromising the general helpfulness of the model. Future work will look to extend these defenses to dynamic modalities like video and audio.

Summary updated

Abstract

Multimodal Large Language Models (MLLMs) achieve strong reasoning and perception capabilities but are increasingly vulnerable to jailbreak attacks. While existing work focuses on explicit attacks, where malicious content resides in a single modality, recent studies reveal implicit attacks, in which benign text and image inputs jointly express unsafe intent. Such joint-modal threats are difficult to detect and remain underexplored, largely due to the scarcity of high-quality implicit data. We propose ImpForge, an automated red-teaming pipeline that leverages reinforcement learning with tailored reward modules to generate diverse implicit samples across 14 domains. Building on this dataset, we further develop CrossGuard, an intent-aware safeguard providing robust and comprehensive defense against both explicit and implicit threats. Extensive experiments across safe and unsafe benchmarks, implicit and explicit attacks, and multiple out-of-domain settings demonstrate that CrossGuard significantly outperforms existing defenses, including advanced MLLMs and guardrails, achieving stronger security while maintaining high utility. This offers a balanced and practical solution for enhancing MLLM robustness against real-world multimodal threats. Our code is released: https://github.com/ZhangXu0963/CrossGuard.
BibTeX
@misc{zhang2025crossguard,
  author = {Xu Zhang and Hao Li and Zhichao Lu},
  title = {{CrossGuard: Safeguarding MLLMs against Joint-Modal Implicit Malicious Attacks}},
  year = {2025},
  eprint = {2510.17687},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.17687},
}
Figure