Models that consume multiple input channels — images plus text, audio plus video — do not use their inputs equally. During training, one modality is usually easier to optimize, so the network leans on it and quietly under-uses the rest; this is modality dominance, and it is one of the best-documented failure modes of multimodal learning. At the same time, attackers who perturb model inputs face a stubborn trade-off: aggressive perturbations reliably flip the model's decision but look anomalous to automated detectors and human reviewers, while subtle perturbations stay unnoticed but often lack the leverage to change anything. This proposal argues that these two problems are two sides of one map. The model's internal modality imbalance tells an attacker exactly where a small distortion buys the largest decision change — and, counterintuitively, also tells the attacker where to spend a small "camouflage" budget on the ignored modality so that the model's internal cross-modal statistics still look healthy to a defender.
The immediate motivation is a new class of defenses that inspect how modalities cooperate rather than how each input looks. FlowGuard, presented in Securing Multimodal AI through Internal Information Decomposition, decomposes a multimodal model's first-token predictions into vision-only, text-only, and joint paths, and flags inputs where fusing the modalities reduces agreement ("negative synergy") — it cuts attack success on LLaVA-1.5-7B from above 90% to under 15%. The premise — adversarial inputs necessarily fracture information flow — has never been tested against an adversary who optimizes against it. Meanwhile, USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models shows that the most potent attacks are precisely those where each modality looks benign in isolation, and When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models shows that attribution machinery (Shapley-style contribution scores) can quantify which stream actually carries an attack's effect. Nobody has combined these: an inference-time attack that steers its budget using the target's measured modality dominance while explicitly preserving the joint fusion signature that consistency-based defenses check.
Core Claim
The claim has three falsifiable parts. First, attack utility per unit of distortion is proportional to a modality's attribution in the target model: because a dominant stream drives the decision, small nudges there flip outputs cheaply. Second, the optimal budget allocation across modalities changes with the defense tier. Against an undefended model, concentrating the entire stealth budget on the dominant modality should be optimal; against a cross-modal consistency detector, the attacker must move budget to the weak modality to keep the synergy signal non-negative — the stealth-utility frontier bends around the detector, and the bending is predictable from measured dominance. Third, balanced-training interventions flatten the dominance gradient, and it is currently unknown whether that makes coordinated attacks more expensive (no cheap single-stream lever) or less (the model now attends to both streams, so smaller perturbations in either one suffice). Assessing the adversarial robustness of multimodal medical AI systems hints at the latter: when only the image stream of a late-fusion chest-X-ray system was attacked at PGD $\epsilon=0.2$, fusion kept an F1 of 83.51 even though the unimodal vision model's F1 fell to 0.0 — but the paper only tried loud, uncoordinated, stealth-free attacks, and its concurrent-attack collapse (F1 between 12% and 31%) came with no stealth accounting at all.
Research Program
The core artifact is a dominance-steered allocation attack, which I'll call DSA. Given a target $f$ taking an image $x$ and text $q$, the attacker chooses perturbations $\delta_x, \delta_q$ to solve
$$\max_{\delta_x,\,\delta_q}\ \mathcal{L}_{\text{atk}}\big(f(x+\delta_x,\;q+\delta_q)\big)\ \ \text{s.t.}\ \ d_v(x,x+\delta_x)\le\epsilon_v,\ \ d_t(q,q+\delta_q)\le\epsilon_t,\ \ S(x+\delta_x,\,q+\delta_q)\ge 0,$$
where $d_v$ and $d_t$ are per-modality stealth distances (KL divergence between attacked and natural per-modality statistics in the spirit of the $\epsilon$-illusory formulation of Illusory Attacks: Information-theoretic detectability matters in adversarial attacks, or the perceptual thresholds SSIM > 0.92 and BERTScore > 0.86 used by AMA: Adaptive Multimodal Adversarial Attack with Dynamic Perturbation Optimization), and $S$ is the partial-information-decomposition synergy of the fused prediction, computed exactly as FlowGuard computes it. The experimental variable is the allocation ratio $\lambda = \epsilon_v/(\epsilon_v + \epsilon_t)$, steered online by per-modality attribution $\phi_v, \phi_t$ — Shapley attribution following When One Modality Rules Them All, or the cheaper cosine similarity between unimodal and fused gradients in the style of MASAM MASAM. The design deliberately reuses the defender's own signal: the attacker keeps $S \ge 0$ not by guessing what looks benign but by measuring the same quantity FlowGuard measures.
This is not a symmetric-channel attack. Cross-Modal Stealth: A Coarse-to-Fine Attack Framework for RGB-T Tracker already builds coordinated perturbations, but for two interchangeable visual channels with no dominance asymmetry and no internal dominance metrics; DSA instead treats a severely imbalanced, text-dominant architecture — the regime that C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models documents in modern multimodal LLMs. Nor is it an adaptive-confidence attack: AMA (above) modulates perturbation strength from output confidence and task difficulty, which says nothing about which stream to spend on and imposes no information-theoretic or joint-signature stealth constraint; DSA differs on exactly those two axes, and AMA is therefore the principal attack baseline.
The second half of the program is a measurement study rather than a method: characterize the stealth-utility frontier — attack success versus per-modality distortion and versus detector evasion — as a function of the target's dominance level, on both standard-trained and balance-corrected models. The balancers come from the dominance literature: Gradient-Blending from What Makes Training Multi-Modal Classification Networks Hard?, which reweights modalities by their overfitting-to-generalization ratios; MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance; and CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery. None of these has ever been evaluated under attack, so the direction of the effect is genuinely open.
Proposed Experiments
The first experiment sweeps the allocation ratio. On CREMA-D audio-visual emotion recognition, I will train standard joint models (the CAT-GS paper reports 67.47% accuracy for standard joint training on CREMA-D; the MMPareto paper reports 71.10% for its uniform baseline) and attack them with $\lambda$ swept from 0 (all budget on the ignored audio stream) to 1 (all on the dominant stream), recording accuracy drop, per-modality distortion, and detection rate under per-modality out-of-distribution detectors. The prediction is a frontier whose optimum sits near $\lambda \propto \phi$ when no cross-modal detector is present. The same sweep runs on Kinetics RGB+audio models, where What Makes Training Multi-Modal Classification Networks Hard? reports 72.6% for RGB-only, 71.7% for naive joint training, and 75.8% for G-Blend, giving dominance-graded targets at matched task.
The second experiment is the defense-evasion test on LLaVA-1.5-7B with FlowGuard deployed, using its own setup (Isolation Forest calibrated on 10,000 benign VQAv2 pairs) and its attack suites (MM-SafetyBench, MOSSBench and the like), plus USB-Base and USB-Hard with its 14,126 and 3,935 samples. Success means materially restoring attack success above the under-15% level FlowGuards achieves against existing attacks — I pre-register ≥50% as a strong result — while per-sample detection stays at the benign false-positive level (the Illusory Attacks paper reports roughly 3% as the natural false-positive rate of its OOD detectors, a comparable reference point). Stealth is held to AMA's published thresholds (SSIM > 0.92, BERTScore > 0.86), and a human study checks that flagged rates match unattacked sequences — the Illusory Attacks paper reports ~32% human flagging for its stealthiest attacks, versus over 80% for classic ones, which is the bar to meet.
The third experiment attacks balanced targets. On CREMA-D I compare robustness of G-Blend, MMPareto (75.13% reported clean accuracy, with its audio encoder alone at 65.46% versus a standalone audio model's 61.69%), and CAT-GS (86.29%) models against single-stream and DSA-coordinated attacks at matched clean accuracy. The medical-system result predicts balanced models shrug off single-stream attacks; the open question is whether coordination then requires larger total budgets (dominance flattened, no cheap lever) or smaller per-modality budgets (both streams now attended). Either outcome is a publishable finding about whether balancing interventions are robustness assets or liabilities.
The fourth experiment is the ablation grid: DSA with the synergy constraint removed (per-modality stealth only), with the dominance steering replaced by uniform allocation, and with steering replaced by AMA-style confidence feedback. This isolates which ingredient does the work against which defense tier.
Baselines
The "do nothing" reference is the undefended target under a uniform equal-budget joint attack — the medical robustness paper's concurrent PGD-plus-synonym-replacement setting, which collapsed multimodal F1 to between 12% and 31% on the Indiana University chest-X-ray data. This baseline establishes that the attacks work at all when no stealth constraint binds, and it shows what "loud" success looks like: large accuracy drops that any reasonable detector catches.
The strongest existing attack baseline is AMA AMA, which reports an attack success rate of 89.56% on ViLT over VQAv2 (versus roughly 78% for its baseline attacks) and 61.82% on BLIP. It is the strongest competitor because it is the only prior attack that adapts its perturbations online to a multimodal target's behavior. It will be run with the same query budget to the target, the same surrogate access, the same compute, and the same stealth thresholds (SSIM > 0.92, BERTScore > 0.86) as DSA, both bare and under FlowGuard. Beating it means higher attack success at matched stealth when no defense is present, and — the decisive comparison — substantially higher success under deployed FlowGuard, where AMA's lack of any joint-signature constraint should get it caught.
The strongest defense baseline is FlowGuard Securing Multimodal AI through Internal Information Decomposition, which reduces attack success on LLaVA-1.5-7B from above 90% to under 15% while keeping benign accuracy within 2.4% of the undefended model and adding about 1.3 seconds of latency. DSA beats it by pushing attack success under the deployed defense well above that under-15% level at benign-level detection rates. Its documented soft spots — a 13–14% residual attack success on pure text jailbreaks and its need for logit access — mark where the evasion experiment should concentrate.
The key ablation baseline is DSA with the synergy constraint $S \ge 0$ deleted but per-modality stealth kept. If that variant evades FlowGuard as well as full DSA, the dominance-steering story is unnecessary and the contribution collapses to "add a synergy penalty to any multimodal attack" — a smaller but still real result, and the experiments are designed so that this outcome would be visible and reportable rather than hidden. The converse ablation, uniform allocation with the synergy constraint kept, plays the same role for the steering ingredient.
Why not just…?
Why not just attack the dominant modality with a standard $L_\infty$ attack? This is the cheapest version of the idea and the most serious threat. It will likely win on an undefended target, and experiment one includes it as the $\lambda=1$ endpoint so the comparison is direct rather than argued. The file's evidence says it loses once any detection tier exists: the Illusory Attacks paper reports classic attacks flagged by automated detectors nearly 100% of the time and by humans over 80% of the time; FlowGuard still catches most text-only jailbreaks (only 13–14% residual success); and the USB benchmark shows attacks carried by a single risky modality are weaker than compositions where each input looks harmless in isolation. If the $\lambda=1$ endpoint matches DSA at equal stealth budgets under FlowGuard, the second hypothesis is falsified and I will report that the frontier does not bend.
Why not just use AMA's adaptive feedback loop and call it a day? AMA already adapts to the target online, so a reviewer could say the machinery exists and only cosmetics differ. But AMA's feedback signal is output confidence on the fused prediction — it cannot see which stream is doing the work, and it carries no constraint on the joint information flow, which is exactly the quantity FlowGuard inspects. It also degrades under surrogate transfer, a known limitation the file flags. AMA is run as the principal baseline under identical budgets, and I am honest about the risk: if adding a synergy penalty to AMA closes the evasion gap, then the contribution narrows to the constraint, not the dominance steering, and the ablation grid is constructed to detect precisely that.
Why not just deploy FlowGuard and skip the attack study? The defense-side version of this proposal — build better detectors, not better attacks — is legitimate, and FlowGuard is a strong start. But its central premise (adversarial inputs produce negative synergy) is an assumption about attacker behavior, not a theorem, and the USB benchmark's finding that safe-image/safe-text compositions are the most effective vectors (with open-source models above 80% attack success) suggests the premise is fragile. A defense whose soundness is untested against an optimizing adversary is exactly the kind of thing red-team work exists to check; this proposal is that check for the consistency-based defense class, and its result — either a working evasion or a survived stress test — is informative in both directions.
Decisive Checks
Three outcomes decide the proposal. First, under deployed FlowGuard on LLaVA-1.5-7B, DSA restores attack success to at least 50% at benign-level detection rates and AMA's published stealth thresholds, against the under-15% level FlowGuard achieves on prior attacks. Second, the human study shows flagged rates indistinguishable from unattacked sequences, with the ~32% illusory-attack flag rate as the reference band. Third, the ablation grid cleanly attributes the effect: the synergy constraint is what evades FlowGuard, and dominance steering is what lowers per-modality budgets at matched success on undefended targets; if either attribution fails, that ingredient is dropped and the finding is reported as a negative result — including the possibility, left open by the balanced-target experiment, that modality balancing is a robustness liability rather than an asset.
References
- Securing Multimodal AI through Internal Information Decomposition
- USB: A Comprehensive and Unified Safety Evaluation Benchmark for Multimodal Large Language Models
- When One Modality Rules Them All: Backdoor Modality Collapse in Multimodal Diffusion Models
- Assessing the adversarial robustness of multimodal medical AI systems: insights into vulnerabilities and modality interactions
- Illusory Attacks: Information-theoretic detectability matters in adversarial attacks
- AMA: Adaptive Multimodal Adversarial Attack with Dynamic Perturbation Optimization
- MASAM: Multimodal Adaptive Sharpness-Aware Minimization for Heterogeneous Data Fusion
- Cross-Modal Stealth: A Coarse-to-Fine Attack Framework for RGB-T Tracker
- C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models
- What Makes Training Multi-Modal Classification Networks Hard?
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
- CAT-GS: Balanced Multimodal Learning via Calibrated Gating and Fusion Surgery