When we align a Large Language Model (LLM) for safety, we typically train it on text. Through techniques like Reinforcement Learning from Human Feedback (RLHF), we teach it to recognize toxic words, dangerous instructions, and harmful requests. However, when these models are extended to process images, audio, or video, this hard-won safety alignment frequently collapses.
This vulnerability is known as the cross-modal safety gap. It is not simply a matter of missing training data; it represents a fundamental structural mismatch between how safety guardrails operate and how multimodal models process information.
Visual and Audio Channels as Unaligned Backdoors
Traditional safety filters operate on discrete linguistic patterns, scanning text inputs for blacklisted keywords or malicious intent. However, multimodal foundation models rely on vision or audio encoders to map high-dimensional, continuous data into a shared embedding space. If safety alignment is not applied uniformly across all input modalities or at the shared embedding level, these non-textual channels act as wide-open backdoors directly into the model's core reasoning engine.
This architectural loophole can be exploited in several ways:
- Adversarial Perturbations: Attackers can apply imperceptible pixel tweaks to an image to hijack the model's output. Christian Schlarmann et al. demonstrated in On the Adversarial Robustness of Multi-Modal Foundation Models that modifying pixels by as little as 1/255th can reliably force a model to generate malicious text strings, such as phishing prompts or misinformation. Because these continuous perturbations bypass discrete text-based safety filters, text-aligned guardrails fail to detect the threat (Xiaosen Wang et al. in Attention! Your Vision Language Model Could Be Maliciously Manipulated).
- Modality Shifting: A model that robustly refuses a harmful text prompt will often execute the exact same instruction if it is embedded within an image or spoken in an audio file. As Erfan Shayegani et al. note in Survey of Vulnerabilities in Large Language Models Revealed by Adversarial Attacks, this occurs because the model's vision and text components often lack a unified safety representation. Similarly, Mintong Kang et al. found in AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models that specialized audio optimization can bypass text-based guardrails by hiding malicious prompts within seemingly benign sounds.
- Cross-Modal Interference: The mere presence of an unaligned modality can degrade existing text safety. Yifan Li et al. discovered in Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models that even a blank image can distract a model from its text-based safety training. This cross-modal interference suggests that integrating visual inputs inherently weakens the linguistic guardrails established during unimodal text alignment (Yihe Fan et al. in Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security).
Information Complementarity and Implicit Reasoning
The cross-modal safety gap widens further when an exploit does not rely on overt malicious instructions, but rather on information complementarity—where an image and a text prompt are completely benign in isolation, but hazardous when combined.
[ Benign Image: A standard household chemical ]
+
[ Benign Text: "How do I safely mix this with bleach?" ]
=
[ Unsafe Output: Instructions for creating toxic gas ]
As detailed by Jingen Qu et al. in Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios and Siyin Wang et al. in Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models, mainstream multimodal models frequently fail to recognize these implicit threats.
An attacker can use cross-modal reference pointers to replace explicit prohibited keywords in a text query with generic references to the visual input (e.g., "Write a step-by-step guide to replicate what is happening in this picture"). This forces the model to synthesize the harmful context internally by combining the benign text with the visual information, bypassing standard keyword-scanning guardrails (Yifan Li et al.).
Because text-only safety filters lack visual grounding, they suffer from high false-negative rates; they cannot identify when a neutral instruction becomes a policy violation (Wei Cai et al. in Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models).
Latent Space Steering in Text-to-Image Generation
The safety gap also operates in reverse, where benign text inputs are used to trigger unsafe visual outputs. Traditional text-to-image safety layers scan prompts for explicit blacklisted words. However, text-to-image models exhibit a cross-modal alignment vulnerability where subtle, non-explicit linguistic cues can successfully steer the model's latent space toward generating restricted content.
The authors of ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users demonstrated that automated feedback loops can easily find prompts that appear benign to text filters but consistently trigger harmful image generation. This occurs because text-based filters fail to account for the complex visual associations models learn during training.
As Xiao Yang explains in BSPA: Exploring Black-box Stealthy Prompt Attacks against Image Generators, an innocent-looking prompt can contain enough latent information to nudge the model into unsafe regions of the generative space. By using indirect language, semantic substitutes, or cultural proxies, attackers exploit the gap between text-only input filtering and the underlying visual synthesis process (Minseon Kim et al. in Automatic Jailbreaking of the Text-to-Image Generative AI Systems).
The Failure of Post-Training Generalization
Why can't we simply patch these vulnerabilities as they arise? The core difficulty lies in the fact that post-training safety alignment does not automatically produce a universal safety layer that transfers across the entire modality space.
Leyi Pan demonstrated in Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models that safety alignment performed on specific modality pairs (such as image and text) frequently fails to generalize to unseen combinations (such as audio and video) or triple-modal inputs.
When a model integrates diverse sensory streams, it introduces unique interactions that are completely absent during unimodal or dual-modal training. Without a unified safety representation that binds all input channels simultaneously, expanding a model's sensory capabilities will inherently continue to expand its adversarial attack surface.