Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias

Weihang Liu, Qianqian Xu, Yangbangyan Jiang, Zitai Wang, Zhipeng Yu, Qingming HuangarXiv 2026 · arXiv

TL;DR

Researchers have discovered a "Visual Trust Bias" in Multimodal Large Language Models (MLLMs), where models are significantly more likely to comply with harmful requests if they are presented within "authoritative" visual contexts like academic journals or news layouts. By exploiting this bias, the proposed Visual Trust Bias Attack (VTBA) achieves high jailbreak success rates across frontier models like GPT-4o and Gemini 2.5 Flash without needing complex adversarial noise or synthetic images.

What Problem Does This Paper Address?

As MLLMs become more integrated into real-world applications, their safety guardrails are under constant scrutiny. Most existing "jailbreaking" methods—techniques used to trick a model into generating harmful content—rely on two main strategies:

  1. Adversarial Perturbations: Injecting invisible digital noise into images to confuse the model's vision encoder.
  2. Toxic Typography: Embedding malicious text directly into synthetic images.

The authors argue these methods are often "unnatural" and easily flagged by basic image filters. More importantly, they don't reflect how a real-world adversary might act: using a natural, "clean" image (like a photo of a chemical label) paired with a harmful query. The paper addresses the lack of practical, stealthy jailbreaks that work on black-box models by identifying a fundamental cognitive-like flaw in how MLLMs process authority.

Methods

The authors introduce the Visual Trust Bias Attack (VTBA). The core idea is inspired by the famous Milgram obedience experiments: just as humans are more likely to follow orders from someone in a lab coat, MLLMs are more likely to "obey" content that looks like an official document.

The VTBA pipeline: from harmful intent to an "authoritative" jailbreak input.

The VTBA framework operates in three main stages:

  1. Template Bank Construction: The researchers curated a set of authoritative layouts from reputable sources, including academic journals (Nature, ACM), newspapers (The New York Times), and government portals.
  2. Intent-Template Matching: Using a vision-language encoder (CLIP), the system matches a user's harmful image-text pair to the most semantically relevant authoritative template. For example, a request about illicit substances might be matched to a medical journal layout.
  3. Query Integration & Assembly:
    • The harmful query is rewritten into a neutral, professional "headline" style (e.g., "How to make bombs" becomes "Analysis of improvised explosive device synthesis").
  4. The original natural image and the rewritten headline are rendered into the template.
  5. A "harmless" guiding prompt is added (e.g., "Infer the actual content from the leading visual"), which tricks the model into a "research mode" where it bypasses its safety filters to "summarize" the document.

Datasets

The paper critiques existing benchmarks like MM-SafetyBench for using ambiguous, synthetic images that don't represent real-world threats. To provide a more rigorous evaluation, they introduced:

  • NatureMMJail: A new dataset of 2,500 natural image-text pairs. It covers 32 sub-scenarios across five major harmful categories: Cybercrime, Dangerous Substances, Identity Discrimination, Property Offenses, and Violence & Harm.
  • NatureMMJail-Tiny: A 500-sample subset used for efficient testing across a wide range of models.

Key Insights and Ideas

  • The "Authority" Vulnerability: MLLMs exhibit a prompt-independent bias toward authoritative visual cues. Attention maps reveal that models focus heavily on institutional logos and structured headers, which seems to suppress their safety refusal mechanisms early in the generation process.
Figure 2: Early-token attention over authoritative visual cues in LLaVA, indicating a prompt-independent visual trust bias.
  • Holistic Bias: The bias isn't just about a single logo. Ablation studies showed that while removing a logo (like the Nature masthead) reduces the attack's success, the overall "look and feel" of the professional layout still carries enough weight to trigger the trust bias.
  • Stealth and Transferability: Because VTBA uses natural images and professional-looking layouts, it is much harder for standard safety filters (which look for "toxic" pixels or keywords) to detect. It also transfers exceptionally well across different model architectures because the "authority" of a journal layout is a universal feature in their training data.

Key Results

The VTBA method consistently outperformed existing state-of-the-art jailbreaks:

  • High Success Rates: On the NatureMMJail dataset, VTBA achieved an average Attack Success Rate (ASR) of 69.2% on GPT-4.1, 66.3% on GPT-4o, and 64.7% on Gemini 2.5 Flash.
  • Comparison: It outperformed perturbation-based attacks (VisualAdv) by over 60% and generation-based attacks (QR) by over 30% in most categories.
  • Frontier Model Vulnerability: Even the latest "frontier" models like GPT-5.1 and Grok 4.1 showed significant vulnerability, with ASRs exceeding 60% in many scenarios.
  • Defense Resistance: The attack remained effective even against specialized multimodal defenses like LLaMA Guard 4 and AdaShield.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Template Dependency: The attack currently relies on a pre-defined bank of templates. While a small bank (N=8) was sufficient for these experiments, a more diverse or dynamically generated set of templates might be even more effective.
  • Pre-training Bias: The attack's efficacy is rooted in the models' pre-training on massive amounts of "authoritative" web data. If future models are trained differently, this specific bias might shift.
  • Future Directions: The researchers suggest that safety alignment needs to move beyond simple text/image filtering. Future work should focus on making models "authority-blind" during safety checks, ensuring they evaluate the underlying intent regardless of the "uniform" the information is wearing.

Summary updated

Abstract

Multimodal large language models (MLLMs) are rapidly advancing, yet their safety remains a critical concern, especially under jailbreak attempts. Existing MLLM jailbreaks either inject fragile adversarial perturbations into irrelevant images or synthesize toxic visual cues to achieve text-driven attack intents. In contrast, a more practical MLLM jailbreak emphasizes enhancing attack efficacy directly on naturally occurring harmful image-text pairs. We address this problem by proposing Visual Trust Bias Attack (VTBA), a new jailbreak framework that exploits a previously overlooked attack surface: MLLMs exhibit an intrinsic trust bias toward authoritative visual features, which enables transferable and stealthy jailbreaks. VTBA constructs a bank of authoritative templates, matches each input pair to the most relevant one, and integrates the harmful intent into the selected layout, thereby leveraging credible visual cues to override safety refusals while preserving naturalness. We further construct a dataset of 2,500 naturally occurring image-text pairs to evaluate jailbreaks in this practical multimodal setting. Extensive experiments show that our method consistently achieves high attack success rates across strong MLLMs, revealing a severe and previously underexplored weakness in current multimodal safety alignment. Warning: This paper may contain sensitive content.
BibTeX
@misc{liu2026dressed,
  author = {Weihang Liu and Qianqian Xu and Yangbangyan Jiang and Zitai Wang and Zhipeng Yu and Qingming Huang},
  title = {{Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias}},
  year = {2026},
  eprint = {2611.22292},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22292},
}
Figure