Vision-language models (systems that read an image and a text question and produce an answer) give different answers when the same question is reworded, when answer options are reordered, when the image placeholder moves within the prompt, or when a chart is redrawn with grid lines. Published accuracy numbers on multimodal math and chart benchmarks average over one fixed phrasing, so they mix two quantities: what the model can reason out, and how lucky that phrasing happened to be. We propose to train a small set of learned input vectors, called a soft prompt, using no answer labels at all. The vectors are trained only to make the model's answer distribution agree across meaning-preserving variants of each input. They are inserted only into the band of layers where probing work says wording first leaks into the computation. If this narrow, label-free intervention removes most of the brittleness, we learn where brittleness enters. Any accuracy gain it produces on reasoning benchmarks measures how much of the so-called multimodal reasoning gap is really instability rather than missing reasoning.
The two-sentence version: a probing study found that multimodal models become sensitive to wording in a specific early-middle band of layers. We test that finding causally by training wording-invariance into exactly that band with a label-free consistency objective, and then measure how much benchmark failure disappears.
Core Claim
Two diagnostic results motivate the claim. The first is the layer-wise probing study How Multimodal LLMs Solve Image Tasks. It trains linear readouts on each layer of LLaVA-1.5 and Qwen2-VL and finds a staged pipeline. Early visual-grounding layers ignore prompt wording. A "lexical integration" band (layers 5–11 in LLaVA-1.5) loses up to 40% probe accuracy when "image" is swapped for "picture". Semantic-decision layers and answer-formatting layers come after. That work is purely diagnostic and uses dog-breed classification. The second is ProSA, which ties sensitivity in text-only models to flat output distributions and shows that dataset averages hide answer flips on individual items. Our hypothesis is that if wording enters at a localizable stage, then making representations invariant at that stage should remove most variant-induced flips. It should do so without teaching the model the task, which is why the training signal carries no labels.
The intervention is a deep soft prompt in the style of P-Tuning v2. That method adds learned vectors at every layer of a frozen model and matches full fine-tuning while training 0.1–3% of parameters. Our variant restricts the vectors to a chosen layer band. For an input $x$ with meaning-preserving variants $v_1,\dots,v_k$, let $p_\theta(\cdot \mid v)$ be the answer distribution of the frozen model with prefix parameters $\theta$. Let $q(\cdot \mid x)$ be the frozen model's own answer distribution averaged over the $k$ variants without any prefix, held fixed during training. We minimize
$$ \mathcal{L}(\theta) = \frac{1}{k}\sum_{i=1}^{k} \mathrm{KL}\big(q(\cdot \mid x)\,\|\,p_\theta(\cdot \mid v_i)\big). $$
The target is the model's own prompt ensemble, so the objective cannot drift toward label-specific knowledge or collapse to a constant answer. In effect it compresses a $k$-times-more-expensive ensemble into a single forward pass. The scientific content lies in the layer restriction and in what happens downstream. The engineering content is the cost reduction.
This differs from the closest prior work in each direction. Misaligned Roles, Misplaced Images shows that moving the image token or swapping role tags raises jailbreak success in Qwen2-VL-7B from near 0% to 42.5%, and repairs this with full adversarial fine-tuning on eight structural variants. We use a frozen backbone, no labels, layer-localized parameters, and a reasoning target rather than a safety target. Prompt Tuning for Unified Multimodal Pretrained Models found that prefix-tuned OFA resists pixel-level adversarial noise better than full fine-tuning. That work supplies the method's plausibility but never tests wording or template variation, and it trains with task labels. Conditional Prompt Tuning for Multimodal Fusion and DePT make soft prompts instance-conditional or decoupled for classification. Both are supervised and address accuracy, not invariance.
Research Program
We first build perturbation-expanded test sets. Following When Benchmarks are Targets and The Butterfly Effect of Altering Prompts, we generate four families of variants. Lexical variants come from synonym and politeness edits and from ProSA's four template categories. Option-order and option-symbol variants come next. Structural variants follow Shayegani et al., with image-token position and role-tag order changed. Rendering variants follow Plot Twist: we redraw synthetic charts with added grid lines, changed marker styles, and stripped titles. Rendering variants matter because they cannot be removed by any text-side rewriting. They also test whether the "prompt-invariant" visual stage is invariant to the image itself.
We then train band-restricted prefixes on unlabeled questions from sources disjoint from the test benchmarks: synthetic plots we render ourselves and the questions of a chart training corpus. We train separate prefixes for four placements. These are the lexical band, the semantic band (layers 12–15 in LLaVA-1.5 and 10–20 in Qwen2-VL), all layers, and the input layer only. Finally, we rerun the probing protocol of the COLM study with and without the prefix. This checks whether the 40% probe drop in the lexical band actually shrinks, or whether invariance is achieved somewhere else.
Benchmark limitations act as the evaluation lens throughout. Every result is reported as canonical-prompt accuracy, mean and worst-case accuracy across variants, instance flip rate, and ProSA's PromptSensiScore. Chart101 results also include collective accuracy, meaning all elementary questions about one chart answered correctly; top models score below 25% on this according to Plot Twist. Agentic transfer is out of scope except for one cheap probe described below. Agent benchmarks such as SPA-BENCH use clean instructions, and paraphrasing their instructions is a natural follow-up only if the static results are positive.
Baselines
The naive reference point is the frozen model under its default prompt, reported alongside its own distribution of scores across variants. This shows how much a single-phrasing number misstates performance. For context, MATH-Vision reports 30.39% accuracy for GPT-4o and 16.97% for the best open model it evaluated (InternVL-Chat-V1-2-Plus). We found no reported result on Lacuna for LLaVA-1.5 or Qwen2-VL-7B on MATH-Vision, so we measure these ourselves.
The strongest existing method is adversarial fine-tuning on structural variants as in Shayegani et al. On AdvBench it reduced Qwen2-VL attack success from 42.5% back to 0%, while VQA-V2 accuracy "remained stable, even showing slight improvements". We found no reported result on Lacuna for that method on reasoning accuracy or flip rate. We re-run it two ways. One uses LoRA, a low-rank weight update, with a trainable-parameter count matched to our prefix. The other uses full fine-tuning. Both see the same unlabeled variant data and the same ensemble-distillation objective, so that only the parameterization differs, and both get the same hyperparameter search budget. Beating it means equal or lower flip rate at equal or higher canonical accuracy on the held-out benchmarks. It also means less drop on unperturbed VQA-V2, which tests whether the frozen backbone preserves general behavior.
The inference-time prompt ensemble is the teacher itself. Our prefix must reach its accuracy within a small margin at one forward pass instead of $k$. If the prefix matched it only at equal compute, the result would be uninteresting.
The key ablation isolates layer placement. The comparison holds parameter count fixed and contrasts the lexical-band prefix with all-layer, semantic-band, and input-only prefixes. A second ablation swaps the ensemble target for gold labels. This separates the effect of invariance from the effect of task supervision.
Why not just…?
Why not just use few-shot examples to anchor the model? ProSA found that moving from zero-shot to one-shot gives the largest single drop in sensitivity, so this is a real competitor. We include one- and three-shot prompting as a baseline, applied to every variant family. Our expectation is that demonstrations will not help on structural and rendering variants. Shayegani et al. show that in-prompt formatting does not protect against role and image-position changes, and demonstrations themselves become another perturbable part of the prompt. If few-shot prompting matches the prefix on all four families, we report that and drop the prefix claim.
Why not just rewrite every input into a canonical template before inference? This is the most serious threat for lexical and structural variants. A deterministic template enforcer plus a paraphrase normalizer could remove them at near-zero cost, and we expect it to do well there. That is why rendering variants are in the design: grid lines, marker styles, and stripped titles cannot be canonicalized in text. Plot Twist reports that adding grid lines alone dropped PaLI-3 from 29.5% to 21.3%. A canonicalizer also ties accuracy to whichever template it chooses. We report the canonicalizer per perturbation family. If it equals the prefix everywhere except rendering, the honest conclusion is that text brittleness is an input-handling problem, and the prefix's contribution narrows to visual-form invariance.
Why not just fine-tune with LoRA on augmented data? This may achieve similar invariance, and it is our strongest baseline. The difference we test is whether layer-localized, frozen-backbone training gives invariance with less damage to unperturbed behavior. OFA-PT's robustness result suggests it might. Evaluating the Impact of Model Scale for Compositional Generalization reports that full fine-tuning distorts pre-trained representations. Even if LoRA ties on invariance, the placement ablation still answers the localization question, which LoRA applied everywhere cannot.
Why not just add chain-of-thought? MATH-Vision found chain-of-thought inconsistent for multimodal models and sometimes harmful, because models hallucinate visual descriptions. We treat a chain-of-thought instruction as one more variant to stay invariant across, not as a fix.
Decisive Checks
Three results would settle the proposal. First, localization: if the lexical-band prefix achieves at least most of the flip-rate reduction of the all-layer prefix at equal parameter count, and the probe drop in that band shrinks, the probing study's stage map is causally confirmed. If the semantic band or the input layer does as well, the map is wrong for this purpose. Second, the gap decomposition: on MATH-Vision and Chart101, compare canonical and worst-case accuracy before and after invariance training. A rise in worst-case accuracy toward the best-variant accuracy, with no labels used, bounds how much of measured "reasoning failure" was instability. A drop in flip rate with flat accuracy says the reasoning gap is genuine, and that MATH-Vision's 42.2% reasoning-error share for GPT-4V is not a phrasing artifact. Third, rendering transfer: the prefix trained on our synthetic chart renderings must reduce flips on Chart101's real-world charts. Collective accuracy is the summary, since a model that answers consistently across one chart's questions shows coherent perception rather than coincidence.
As a single low-cost agentic probe, we paraphrase instructions for a small set of single-app SPA-BENCH tasks. We check whether a prefix trained only on static variants changes success-rate variance across paraphrases. This tells us whether the follow-up on agent benchmarks is worth its cost, about $0.09 per step for M3A according to SPA-BENCH.
References
- How Multimodal LLMs Solve Image Tasks: A Lens on Visual Grounding, Task Reasoning, and Answer Decoding
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks
- Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots
- Prompt Tuning for Unified Multimodal Pretrained Models
- Conditional Prompt Tuning for Multimodal Fusion
- DePT: Decoupled Prompt Tuning
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- SPA-BENCH: A COMPREHENSIVE BENCHMARK FOR SMARTPHONE AGENT EVALUATION
- Measuring Multimodal Mathematical Reasoning with MATH-Vision Dataset
- Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing