FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models

Jiaming Zhang, Zherui Li, Siqi Guo, Junhao Dong, Kun Wang, Wei Yang Bryan LimarXiv 2026 · arXiv

TL;DR

FusionAudit introduces a diagnostic-driven framework to uncover hidden vulnerabilities in modern "native" Vision-Language Models (VLMs) by targeting specific visual pathways. By moving beyond generic adversarial attacks to architecture-aware interventions like MANTIS and Nat-Evid, the authors demonstrate that standard benchmarks significantly underestimate the residual risk in deeply integrated multimodal systems.

What Problem Does This Paper Address?

The architecture of Vision-Language Models is shifting. Early "alignment-era" models (like LLaVA) were modular, essentially piping a frozen vision encoder into a language model via a simple projection layer. Modern "native" VLMs, however, feature deep fusion where visual and textual tokens interact in a unified representational space from the very first layers.

Current robustness evaluations haven't kept pace with this structural transition. They typically rely on "pixel-centric" attacks (like PGD) that treat the entire visual input as a single surface. This approach fails for native models because:

  1. Gradient Mismatch: Standard end-to-end attacks spread gradient pressure across all output tokens, often missing the specific "decision token" where a model commits to an error.
  2. Visual Rejection: Native models exhibit a "see-but-not-deceived" behavior—they might "read" an adversarial text overlay but "reject" it as a task response because it looks visually incongruent with the rest of the image.

Methods

The authors propose PaRMA (Pathway-conditioned Risk for Multimodal Auditing), a framework that decomposes the visual attack surface into two primary pathways: pixel perturbation ($\pi_{pix}$) and rendered-text insertion ($\pi_{txt}$).

Figure 1: The PaRMA framework and FusionAudit pipeline. Native VLMs are evaluated over two visual pathways: pixel perturbation and rendered-text insertion. Baseline attacks (Vis-PGD, End-PGD, Typo-Evid) yield initial failures and diagnostic readouts. These diagnostics surface architecture- associated failure signatures (e.g., the see-but-not-deceived signature), guiding the generation of diagnostic-motivated augmented attacks (MANTIS, Nat-Evid) to compute the final falsifiable residual risk bR+.

To exploit the blind spots of native architectures, they introduce two novel attacks within their FusionAudit pipeline:

  • MANTIS (Margin-Aware Native Token Intervention Strategy): Instead of optimizing against the full output sequence, MANTIS concentrates gradient pressure exclusively on the "first-divergence token"—the exact point where the correct and wrong answer sequences first differ.
  • Nat-Evid (Naturalized Evidence): To bypass the model's visual rejection of simple typographic overlays, Nat-Evid uses generative image editing to insert "host-congruent" text (e.g., making a wrong answer look like a native chart annotation or scene text).

They also introduce two key diagnostics:

  • Read–Use Gap (RUG): Measures the difference between a model's ability to read inserted text and its tendency to use it in a task response.
  • Overlay Attention Ratio (OAR): Quantifies how much attention the model's decoder pays to an adversarial overlay relative to its size.

Datasets

The audit was conducted on a balanced 1500-sample VQA evaluation set compiled from three major benchmarks:

  • ChartQA: Focuses on visual reasoning over structured data.
  • DocVQA: Targets document understanding.
  • MME: Provides a mix of natural image categories (artwork, celebrities) and text-dense reasoning.

The authors evaluated 18 open-weight VLMs categorized by a new "nativeness taxonomy" ranging from $L_1$ (modular) to $L_3$ (deep/early fusion).

Key Insights and Ideas

  • The "See-But-Not-Deceived" Signature: In native VLMs, attention diagnostics (OAR) show that models often focus heavily on typographic overlays, and reading prompts prove they can "see" the text. However, they frequently ignore this text when answering the main task. This suggests native models have a fine-grained sensitivity to visual congruence that modular models lack.
  • Objective Mismatch in Deep Fusion: In $L_3$ models, visual features re-enter the computation at multiple layers. Standard attacks anchored only at the vision encoder or spread across the whole output sequence are less effective than MANTIS, which targets the specific decision-making token.
  • Ranking Instability: The paper reveals that legacy rankings based on standard PGD are unstable for native models. A model that appears robust under PGD might be highly vulnerable to MANTIS, meaning single-scalar benchmarks are no longer reliable for model selection.

Key Results

The diagnostic-driven attacks in FusionAudit significantly raised the estimated residual risk across all model types:

  • Increased Risk Detection: Compared to baseline attacks, the diagnostic-driven suite increased the detected risk by +10.6% in $L_1$, +16.9% in $L_2$, and a staggering +25.1% in $L_3$ models.
  • MANTIS Superiority: The fraction of examples broken exclusively by MANTIS (and missed by standard PGD) peaked at 24.6% for the most advanced $L_3$ models.
  • Stability: Ablation studies showed that MANTIS is robust to hyperparameter choices like random initialization and optimization budget, confirming that its gains are structural rather than just "lucky" tuning.
Figure 2: MANTIS Stability and Ablation. Evaluated on a balanced 300-sample subset across all 18 models. (a) Min-max ASR variance across three seeds, centered on the per-model mean. (b) Clean-correct ASR as PGD steps increase at ε = 8/255. (c) Clean-correct ASR across varying ℓ∞ radii at t = 40. The rightmost bars in each panel show the 18-model mean.

Limitations and Future Work

  • Observational Taxonomy: The $L_1$–$L_3$ levels are correlational; while they track fusion depth, other factors like training data and scale also vary between model families.
  • The $L_4$ Gap: The authors define a theoretical $L_4$ level (strict single-backbone models like Chameleon), but these were not included in the empirical audit due to current limitations in handling text-heavy images.
  • Scope of Attacks: The current framework focuses on correctness-based Attack Success Rate (ASR) for wrong-answer injection. It does not yet cover "jailbreaking" or safety-alignment bypasses, which require different, compliance-based metrics.
  • Extensibility: The authors envision expanding PaRMA to other modalities like audio and video as native multimodal models continue to evolve.

Summary updated

Abstract

As vision-language models (VLMs) transition from frozen-encoder pipelines to native multimodal architectures, the visual channel increasingly serves as both a perceptual input and a carrier of rendered text that competes with user prompts. Current single-attack, pixel-centric evaluations fail to capture this complexity: they collapse distinct visual attack surfaces into a single scalar and ignore how architectural shifts alter vulnerability. We introduce PaRMA, an evidence-based auditing framework that defines pathway-conditioned risk over two primary image-level interventions: pixel perturbation and rendered-text insertion. Instead of arbitrary attack enumeration, PaRMA uses pathway-specific diagnostics to expose architecture-specific blind spots. Notably, we uncover a see-but-not-deceived signature where native VLMs attend heavily to typographic overlays but visually reject them, and we identify an objective mismatch between encoder-level and answer-level attacks in deep-fusion models. To exploit these findings, we instantiate FusionAudit, introducing two diagnostic-motivated attacks: MANTIS (margin-aware pixel perturbation) and Nat-Evid (host-congruent semantic insertion). Evaluating 18 open-weight VLMs across a novel nativeness taxonomy, we demonstrate that legacy single-attack rankings (e.g., PGD) are unstable for native models. Our diagnostic-driven attacks substantially raise the estimated residual risk, showing that PaRMA provides a falsifiable, evidence-driven auditing methodology for uncovering vulnerabilities missed by standard benchmarks.
BibTeX
@misc{zhang2026fusionaudit,
  author = {Jiaming Zhang and Zherui Li and Siqi Guo and Junhao Dong and Kun Wang and Wei Yang Bryan Lim},
  title = {{FusionAudit: Pathway-Conditioned Robustness Auditing for Native Multimodal Models}},
  year = {2026},
  eprint = {2610.21667},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.21667},
}
Figure