The Cross Modal Safety Gap in Multimodal Models

2,945 concepts1,113 papers24 proposals

When we align a Large Language Model (LLM) for safety, we typically train it on text. Through techniques like Reinforcement Learning from Human Feedback (RLHF), we teach it to recognize toxic words, dangerous instructions, and harmful requests. However, when these models are extended to process images, audio, or video, this hard-won safety alignment frequently collapses.

This vulnerability is known as the cross-modal safety gap. It is not simply a matter of missing training data; it represents a fundamental structural mismatch between how safety guardrails operate and how multimodal models process information.

Visual and Audio Channels as Unaligned Backdoors

Traditional safety filters operate on discrete linguistic patterns, scanning text inputs for blacklisted keywords or malicious intent. However, multimodal foundation models rely on vision or audio encoders to map high-dimensional, continuous data into a shared embedding space. If safety alignment is not applied uniformly across all input modalities or at the shared embedding level, these non-textual channels act as wide-open backdoors directly into the model's core reasoning engine.

This architectural loophole can be exploited in several ways:

Information Complementarity and Implicit Reasoning

The cross-modal safety gap widens further when an exploit does not rely on overt malicious instructions, but rather on information complementarity—where an image and a text prompt are completely benign in isolation, but hazardous when combined.

[ Benign Image: A standard household chemical ] 
                       +
[ Benign Text: "How do I safely mix this with bleach?" ]
                       =
[ Unsafe Output: Instructions for creating toxic gas ]

As detailed by Jingen Qu et al. in Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios and Siyin Wang et al. in Safe Inputs but Unsafe Output: Benchmarking Cross-modality Safety Alignment of Large Vision-Language Models, mainstream multimodal models frequently fail to recognize these implicit threats.

An attacker can use cross-modal reference pointers to replace explicit prohibited keywords in a text query with generic references to the visual input (e.g., "Write a step-by-step guide to replicate what is happening in this picture"). This forces the model to synthesize the harmful context internally by combining the benign text with the visual information, bypassing standard keyword-scanning guardrails (Yifan Li et al.).

Because text-only safety filters lack visual grounding, they suffer from high false-negative rates; they cannot identify when a neutral instruction becomes a policy violation (Wei Cai et al. in Safe Semantics, Unsafe Interpretations: Tackling Implicit Reasoning Safety in Large Vision-Language Models).

Latent Space Steering in Text-to-Image Generation

The safety gap also operates in reverse, where benign text inputs are used to trigger unsafe visual outputs. Traditional text-to-image safety layers scan prompts for explicit blacklisted words. However, text-to-image models exhibit a cross-modal alignment vulnerability where subtle, non-explicit linguistic cues can successfully steer the model's latent space toward generating restricted content.

The authors of ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users demonstrated that automated feedback loops can easily find prompts that appear benign to text filters but consistently trigger harmful image generation. This occurs because text-based filters fail to account for the complex visual associations models learn during training.

As Xiao Yang explains in BSPA: Exploring Black-box Stealthy Prompt Attacks against Image Generators, an innocent-looking prompt can contain enough latent information to nudge the model into unsafe regions of the generative space. By using indirect language, semantic substitutes, or cultural proxies, attackers exploit the gap between text-only input filtering and the underlying visual synthesis process (Minseon Kim et al. in Automatic Jailbreaking of the Text-to-Image Generative AI Systems).

The Failure of Post-Training Generalization

Why can't we simply patch these vulnerabilities as they arise? The core difficulty lies in the fact that post-training safety alignment does not automatically produce a universal safety layer that transfers across the entire modality space.

Leyi Pan demonstrated in Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Models that safety alignment performed on specific modality pairs (such as image and text) frequently fails to generalize to unseen combinations (such as audio and video) or triple-modal inputs.

When a model integrates diverse sensory streams, it introduces unique interactions that are completely absent during unimodal or dual-modal training. Without a unified safety representation that binds all input channels simultaneously, expanding a model's sensory capabilities will inherently continue to expand its adversarial attack surface.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multimodal_safety_gap_05206db3,
  title = {multimodal safety gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-safety-gap/rdr_05206db31be7418ebc9d2daa6a033078},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_multimodal_safety_gap_05206db3_v1,
  title = {multimodal safety gap},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multimodal-safety-gap/rdr_05206db31be7418ebc9d2daa6a033078/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake