Anchored Indirect Comparison for Auditing Multimodal Jailbreak Defenses

Papers on defending vision-language models against jailbreaks, meaning inputs that trick a model into producing content it should refuse, each report their defense against an undefended model. They do this on their own choice of victim model, attack and benchmark. Readers then compare defenses across papers by setting these numbers side by side, much as one might compare two drugs by reading their separate trials against placebo. Medicine has a mature tool for that situation, network meta-analysis (NMA). NMA treats every trial as an edge in a graph of treatments and estimates how two treatments compare even when no trial tested them head to head. It does this by chaining each one's effect relative to a shared comparator, and it has explicit checks for when that chaining is not valid. This proposal asks whether NMA can turn the fragmented multimodal-defense literature into rankings that are calibrated and that flag their own failures. Our own experiments show that ranking flips across attack types can make naive cross-paper comparisons misleading. Because ML artifacts can be re-run, we can check the indirect estimates against real head-to-head runs.

The core claim is concrete and falsifiable. Express each defense's effect as a change in the log-odds of attack success relative to the undefended model within the same paper's setting, and adjust for a small set of effect modifiers such as attack modality, model openness and prompt type. Under those conditions, indirect estimates of defense-versus-defense differences track directly measured differences closely, and their confidence intervals cover the truth at close to the nominal rate. The NMA inconsistency diagnostics, which compare direct and indirect estimates around closed loops, should catch the cases where they do not. If this holds, anyone can audit whether claims like "defense X beats defense Y" survive once differences in setting are accounted for.

Core Claim

The transfer from NMA is anchoring. A critical assessment of matching-adjusted indirect comparisons in relation to target populations shows why it matters and where it breaks. Comparing drugs only through their effects relative to a shared control keeps each trial's internal randomization intact. Even so, when an effect modifier is unevenly distributed across trials, the same data can say drug A beats drug B (log odds ratio 0.425) or the reverse (−0.402), depending on which population serves as the target. That paper is the methodological warning this proposal is built around, not a technique we reuse wholesale.

The multimodal literature already contains the raw material for anchored comparisons, but it is used unanchored. BlueSuffix, which combines diffusion image purification, text rewriting and a reinforcement-learned defensive suffix, reports Bi-Modal Adversarial Prompt (BAP) attack success on LLaVA dropping from 61.02% to 4.65%. That is roughly 3.47 in log-odds, by our arithmetic on the reported numbers. MM-SafetyBench reports a safety system prompt dropping LLaVA-1.5-7B success from 77.33% to 15.68% on its tiny split, roughly 2.91 in log-odds. Reading these side by side suggests one defense is better. However, the attacks differ: optimized bimodal prompts in one case, typographic and generated images in the other.

B-AVIBench shows that robustness rankings of models flip with attack type. MiniGPT-4 was the most corruption-resistant model but highly vulnerable to decision-based image attacks, and OpenFlamingo-V2 resisted image attacks but was the most vulnerable to text attacks. Attack modality is therefore a plausible effect modifier, exactly the situation in which unadjusted indirect comparison misleads.

Formally, let $s$ index a "study" (one paper's victim model × attack × benchmark cell) and $k$ index a defense. Let $p_{sk}$ be the jailbreak rate. We fit

$$\operatorname{logit} p_{sk} = \alpha_s + d_k + \beta_k^\top x_s + u_{sk},$$

where $\alpha_s$ is the study's baseline vulnerability (absorbed by anchoring to its own undefended arm), $d_k$ is defense $k$'s effect relative to no defense, $x_s$ are study-level modifiers, $\beta_k$ is how defense $k$'s effect varies with them, and $u_{sk}$ is a random effect for heterogeneity between studies. The quantity of interest is the indirect contrast $d_j - d_k$ evaluated at a chosen modifier profile $x$.

Research Program

The work has three parts.

The first part builds a ground-truth factorial with the OmniSafeBench-MM toolbox. Its benchmark (OmniSafeBench-MM) ships 13 attacks and 15 defenses across 9 risk domains and three prompt types. We run every defense against every attack on four to six open vision-language models, including LLaVA-1.5 and MiniGPT-4 variants. A jailbreak is scored with the benchmark's Harmfulness–Alignment–Detail protocol, which counts a jailbreak when harmfulness is at least 3 and intent alignment is at least 3. We add benign utility on AlpacaEval as a second endpoint, as BlueSuffix reports. This dense table is the ground truth that the medical setting never has.

The second part simulates literature-like sparsity. From the dense table we sample sparse evidence networks that mimic how papers are actually written. Each simulated "paper" tests one new defense against no defense, and sometimes against one or two older defenses, on one or two models and attacks of its choosing. The sampling over-represents popular models and attacks, as real papers do. We fit the anchored NMA, with and without modifier terms, and compare its estimates of held-out head-to-head contrasts to the dense truth. Adding a second endpoint lets us also test multi-outcome NMA, jointly modeling jailbreak reduction and utility loss, the way clinical NMAs weigh efficacy against safety (Comparative efficacy and safety of prostacyclin therapies for pulmonary arterial hypertension).

The third part applies the method to the real literature. We extract undefended and defended arms from published defense papers, including BlueSuffix, MM-SafetyBench's safety prompt, and the off-model filters evaluated in OmniSafeBench-MM. From these we build the actual evidence graph and report which published cross-paper superiority claims survive anchoring and modifier adjustment. Where we can reproduce a paper's setting with the toolbox, we also check the indirect conclusion against a direct run.

Several neighboring lines of work do something different. The closest attack-side works aggregate across settings. DDGA pools gradients across surrogate models to make attacks transfer, and Multimodal Generative Engine Optimization manipulates the rankings that vision-language models produce. Neither estimates relative effects through a shared comparator, nor tests whether such estimates are consistent. Graph-robustness work such as Robustness in Text-Attributed Graph Learning attacks models that run on graphs, which is a different use of the word "network." Diagnostic-test NMA (NMADTA) motivates the multi-endpoint extension. Composite-likelihood NMA (Network meta-analysis made simple) supplies a cheap estimator for when the number of studies is large.

Baselines

The naive reference point is the unanchored cross-paper leaderboard: rank defenses by their reported defended success rate. We implement it on exactly the same simulated sparse networks and the same extracted literature data. Beating it requires a lower held-out error on defense-versus-defense contrasts and a higher rank correlation with the dense ground truth. Showing that it ranks some pairs wrong, as the BlueSuffix-versus-safety-prompt example suggests it might, is the first expected result.

The strongest ML competitor is low-rank tensor completion of the model × attack × defense success table. It is the standard way to fill missing benchmark cells and does not need an anchoring assumption. We give it the same observed cells and the same rank and regularization search budget, chosen by cross-validation on observed cells only. No reported result was found on Lacuna for completion or NMA on any multimodal robustness table, so there is no published number to beat. The comparison is internal to our ground truth. NMA beats completion if it has lower held-out contrast error, or matches that error with intervals whose coverage is near 95% where completion's are not.

The most important ablation is NMA without anchoring. We fit the same random-effects model to raw defended rates instead of within-study contrasts, which isolates whether the shared comparator is what produces the gains. A second ablation drops the modifier term $\beta_k^\top x_s$, which isolates the value of adjusting for attack modality and model type.

For the published-literature part, the strongest reported effects are BlueSuffix's 61.02% to 4.65% on LLaVA under BAP and its reduction to below 0.5% on Gemini 1.5 Flash. Our direct reproduction must recover these to within sampling error before indirect estimates involving BlueSuffix are trusted.

Why not just…?

Why not just run every defense against every attack directly? For open models this is possible, and we do it, because it is the ground truth. The approach falls short in two places. Closed models such as GPT-4V or Gemini do not allow white-box attacks, and gradient-coordinated attacks like Coordinated Robustness Evaluation Framework for Vision-Language Models cannot be run against them, so their cells stay sparse. B-AVIBench needed 316,000 instructions for 16 models, so the full cross-product grows quickly. More fundamentally, direct runs do not tell you whether the existing literature's claims hold. The audit in the third part is about published evidence, and that evidence is sparse by construction. If the experiments show the indirect estimates are useless even when the transitivity checks pass, the honest conclusion is "always run directly," and we would report it.

Why not just fill in the missing cells with matrix or tensor completion? This is the real threat. If a low-rank structure explains the success table well, completion may predict missing contrasts as accurately as NMA without any anchoring assumption. We expect NMA's advantages to come from three places. First, its calibrated uncertainty under few studies. Second, its explicit loop-inconsistency test, which completion lacks. Third, its use of within-paper contrasts, which are robust to judge and prompt differences between papers that shift every arm of a paper together. Completion has no natural way to absorb such shifts without a per-paper offset, and adding one essentially rebuilds anchoring. The first-versus-second-part comparison settles the question. If completion with per-paper offsets matches NMA on both error and coverage, we will report the contribution as the inconsistency diagnostics and the modifier analysis, not as better point estimates.

Why not just use a single standardized benchmark leaderboard? Leaderboards answer a different question, and Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness? shows how a single protocol can mislead. In that study AutoAttack's 100% success on CIFAR-10 fell to 0.33% against a simple frequency-domain detector, while Carlini–Wagner kept over 40%. A leaderboard ranks under one fixed attack mix. Our modifier model asks how a defense's effect changes as the attack mix changes, which is exactly what a single scalar hides. A leaderboard is also only as current as its maintainers, while the published literature keeps growing around it.

Proposed Experiments and Decisive Checks

The primary metric is error on held-out defense-versus-defense log-odds contrasts in the simulated sparse networks, together with the coverage of 95% intervals, the rate at which the sign of a contrast is wrong, and Spearman correlation of the defense ranking with the dense truth. Results are reported as a function of network sparsity, from about 10% to 50% of cells observed.

The first decisive check is a planted modifier. In the dense table we identify defenses whose effect clearly differs between typographic attacks (FigStep-style, from FigStep) and perturbation-based attacks. We then sample networks in which the two defenses were tested on different attack types. The modifier-adjusted model should recover the correct ordering at each attack profile. The unadjusted model should show a MAIC-style flip, and the inconsistency test should flag the affected loops. If the inconsistency test fails to detect planted violations at a useful rate, the method is not fit for auditing.

The second check uses the safety-utility endpoint. Joint modeling of jailbreak reduction and benign utility should identify defenses whose ranking changes once utility is weighed, testing the trade-off that BlueSuffix and OmniSafeBench-MM raise.

The third check applies the method to real published claims. For every pair of defenses whose published comparison is indirect and whose setting we can reproduce, we run the head-to-head directly. We then report how often the naive reading, the anchored-NMA reading and the direct result agree. The proposal succeeds if anchored NMA agrees with direct runs clearly more often than the naive reading and flags the disagreements in advance. It fails if it does no better than tensor completion with per-paper offsets.