When a vision-language model sees the phrase "spill the beans," it has to choose between two meanings: beans hitting the floor, or a secret coming out. Published systems disagree about which way models lean. One team says vision-language models are stuck on the literal reading. Another says large language models over-read figurative intent into phrases that are meant literally. We propose to treat idiom interpretation as a knowledge conflict, the well-studied situation where evidence in the input contradicts what the model memorized during training, and to measure it in both directions. To do this we will build a small but carefully balanced dataset with human verification. In it, the same phrase appears in literal and figurative contexts. The disambiguating evidence comes either from the sentence or from the image.
The central bet is that the "literal versus figurative bias" seen in prior work is not a fixed property of models. It should be predictable from two quantities: how strongly the model reads the phrase figuratively when given no context (its prior), and model size. Benchmarks of factual conflicts already show larger multimodal models becoming more stubborn. If that carries over to idioms, larger models should increasingly force a figurative reading onto frequent idioms even when the sentence or image clearly shows literal use, while still failing to find figurative readings for rare ones. We will test this at 3B to 72B parameters. We will locate where in the network the arbitration happens, and check whether a fix found at small scale carries over to large models.
Core Claim
Define the bias index of a model on a set of phrases as
$$B = P(\text{figurative answer} \mid \text{evidence says literal}) - P(\text{literal answer} \mid \text{evidence says figurative}),$$
where the probabilities are measured on conflict items, meaning items in which the evidence disagrees with the model's own context-free reading.
The claim has three parts. First, $B$ changes sign depending on the strength of the context-free prior: phrases the model strongly reads as figurative give $B>0$, and weakly figurative phrases give $B<0$. This reconciles the opposing reports of AIMA (literal bias) and PALI-NLP / MIRA (figurative bias). Second, the slope of $B$ against prior strength steepens with model size, as the stubbornness trend in MMKC-Bench would predict. Third, the internal signal that decides the arbitration sits at a narrow relative depth at 7B but spreads over more layers at 72B. If so, a correction vector placed at a fixed layer that works at 7B will degrade when moved to 72B by relative depth, and the layer will need to be re-located.
This is new because no prior work measures idiom arbitration in both directions under controlled evidence, or across scale. MMKC-Bench measures stubbornness across scale but only for factual entities and gestures. Pixels Versus Priors locates a mid-network switch from language priors to visual evidence, but only for colour and size. AdMIRe-style systems score only final outputs.
Research Program
Dataset. The first piece is a counterbalanced dataset, which we call Bidirectional Idiom Conflict (BIC). Seed compounds come from the English and Portuguese idiom lists of AdMIRe and from the 223 idioms of the AStitchInLanguageModels dataset that AIMA used for classifier training. Each compound gets four kinds of item.
In the text-evidence condition, a sentence settles the reading in one direction or the other, and the model must rank five candidate images in AdMIRe's format. Every compound has both a literal and a figurative sentence over the same image set. A model that always picks the literal image therefore cannot score above half.
In the image-evidence condition, the sentence is left ambiguous ("At dinner, she spilled the beans") and a scene image settles the reading. The model then chooses between a literal and a figurative paraphrase. This is the multimodal analogue of the context-versus-memory setup in MMKC-Bench and Pixels Versus Priors.
Generation follows the author-reviewer pattern. Native speakers write scene descriptions. A text-to-image model renders them, as AdMIRe did with Midjourney. A segmentation-guided edit, in the spirit of the SAM2 edits in Pixels Versus Priors, produces near-identical literal and figurative scene pairs where possible. Two human verifiers then accept, reject, or minimally edit each item. Minimal editing serves to strip lexical giveaways, the same reason FLUTE used it. Putting humans in a verifying role rather than an authoring role is what makes a few hundred compounds affordable; Bae et al. report review to be about 13 times cheaper than writing from scratch. We target roughly 400 compounds and about 1,600 verified items.
We will audit the dataset for shortcuts. A text-only model and an image-only CLIP similarity scorer should sit near chance on the balanced split. This matters because AdMIRe's existing test set seems exploitable: AIMA's literal-only cosine baseline scored 87% Rank-1 on the test set, above their full mixture-of-experts system at 65%, yet only 48% on the extended set.
Measurements. The second piece is the measurement suite, run on Qwen2.5-VL at 3B, 7B, 32B, and 72B, and on the InternVL3 family as a second model line. For each model we estimate each compound's prior by asking, with no disambiguating context, which paraphrase is meant. We then compute $B$ within prior-strength bins.
Next we probe the internal state. Following Analysing the Residual Stream, which found conflict detectable near layer 14 and the final choice predictable by layers 16–17 in 8B text models, we fit linear probes on the residual stream at the compound's last token. One probe detects conflict, one predicts the final choice. We report the peak layer as a fraction of total depth, and a spread measure: the number of layers whose probe AUROC falls within 0.02 of the peak. We also run a logit-lens analysis, which decodes intermediate layers directly into words, to test whether the "mid-pass flip" from Pixels Versus Priors appears for idioms.
Steering. The third piece is a two-directional steering test. Steering here means adding a fixed vector to the model's hidden state at one layer to push its behaviour. Following the contrastive construction of ContextFocus and the PvP vectors, we build two vectors from items where evidence and prior agree. One pushes toward the evidence when the evidence is literal; the other pushes toward the evidence when it is figurative.
Pixels Versus Priors found it much easier to steer away from a prior toward pixels than to bring a suppressed prior back. That predicts a specific asymmetry here: correcting a figurative over-reading using a literal image should be easier than recovering a figurative reading from a literal-looking scene. The scale test is the one motivated by SOMF, which found functional regions concentrated early at 7B but diffuse in larger models. We pick the layer at 7B, move it to 72B by relative depth, and compare against re-locating the best layer directly at 72B.
Baselines
The do-nothing reference is each model answering zero-shot with a plain instruction. This gives the raw $B$ curve that every intervention must improve. We also report the trivial literal-matching scorer, CLIP cosine similarity between sentence and image. AIMA reports 87% Rank-1 on the AdMIRe test set and 48% on the extended set for this scorer. On BIC's balanced split it should fall to about half, which confirms the counterbalancing works.
The strongest existing method is MIRA from PALI-NLP. It uses in-context literal-versus-figurative examples, refines image captions, infers the latent meaning, and aggregates several self-consistent answers, all without weight updates. It reports 0.93 in AdMIRe Subtask A (English, text and images) and 1.0 sentence-type accuracy on Subtask B. It is the strongest competitor because it targets exactly the bias we measure, and it does so cheaply. We will reimplement it on the same Qwen2.5-VL and InternVL3 backbones, with the same pool of in-context examples drawn from a held-out split of BIC and the same number of sampled passes. We will tune it on the same development split we use to pick steering layers and multipliers.
Beating MIRA means one of two things. Either the steered model gets $|B|$ closer to zero in the high-prior bins at 72B, with confidence intervals that do not overlap MIRA's, or it matches MIRA with a single forward pass. A tie at multi-pass cost would not count as beating it.
We also include AIMA's two-expert router as a weaker trained baseline, trained only on non-test compounds. As a reference for fine-tuning, we run a parameter-efficient fine-tune (low-rank adapters) on the same verified training items.
Two ablations isolate the key ingredients. The first uses a one-directional steering vector, built PvP-style from items where evidence beats the prior regardless of reading direction. This tests whether separating the literal and figurative directions matters. The second uses the 7B-chosen layer transferred by relative depth versus the layer re-located at 72B. This tests the claim about architectural scaling.
Why not just…?
Why not just prompt with balanced in-context examples, as MIRA does? This is the real threat. MIRA's AdMIRe results show that prompting can largely remove bias on that benchmark. Against it, MMKC-Bench found that models detect 75–79% of factual conflicts yet still follow memory, and Pixels Versus Priors reports that steering moves attention to image tokens far more than editing the prompt does. Neither result involves idioms, though. Our experiment settles it directly. If MIRA flattens the $B$-versus-prior curve at every scale, including high-prior compounds at 72B, then the steering component is unnecessary. In that case the contribution becomes the dataset and the diagnostic scaling law, and we will report it that way.
Why not just generate the dataset fully automatically? Unverified figurative generation drifts toward trivial items and lexical shortcuts, which is why FLUTE added human editing. Generated images also frequently fail to depict the intended reading. We will measure this directly. Verifiers will log the rejection rate for raw generated images. We will also train and evaluate on an unverified twin of BIC and check whether the text-only and image-only shortcut baselines rise above chance on it.
Why not just fine-tune on idiom data? Multimodal idiom data is tiny: AdMIRe Subtask A has 102 test instances and Subtask B has 20. SOMF shows that task fine-tuning can damage existing behaviour; harmlessness dropped to 37.19% on CATQA after standard fine-tuning. Fine-tuning also does not tell us why the bias exists. We still run the adapter fine-tune, and we report its $B$ curve together with general-capability retention.
Why not skip internal probing and just report output accuracy? Output accuracy alone cannot test whether a layer chosen at small scale transfers to large scale, and that is the practical question for anyone using steering on 70B-class models. The probe spread measure and the layer-transfer ablation are what make the scaling claim testable.
Proposed Experiments and Decisive Checks
The first check is whether the reconciliation holds. Regress $B$ on context-free prior strength for each model. The claim fails if $B$ keeps the same sign across prior bins for every model, since that would mean the literal/figurative disagreement in the literature comes from task format, not prior strength. We also run AdMIRe's original test and extended splits as an external check, so our numbers can be compared with the published 0.93 and 65%.
The second check is scale. The claim predicts the slope of $B$ against prior strength rises monotonically from 3B to 72B in both model families. The probe spread should widen with size. The transferred-layer steering vector should lose a measurable share of its 7B effect at 72B, while the re-located vector recovers it.
The third check is the asymmetry. Pixels Versus Priors predicts that a literal image corrects figurative over-reading more easily than a figurative scene overrides a literal prior. If we observe the opposite, the result would separate how idioms behave from the colour-and-size findings in that paper.
All results will be reported separately for English and Portuguese. This is a partial check on the Western bias documented by Cultural Bias Matters. Any claim that holds only for English will be stated as such.
References
- AIMA at SemEval-2025 Task 1: Bridging Text and Image for Idiomatic Knowledge Extraction via Mixture of Experts
- PALI-NLP at SemEval 2025 Task 1: Multimodal Idiom Recognition and Alignment
- Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models
- Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts
- SemEval-2025 Task 1: AdMIRe - Advancing Multimodal Idiomaticity Representation
- FLUTE: Figurative Language Understanding through Textual Explanations
- Building a Role Specified Open-Domain Dialogue System Leveraging Large-Scale Language Models
- Analysing the Residual Stream of Language Models Under Knowledge Conflicts
- ContextFocus: Activation Steering for Contextual Faithfulness in Large Language Models
- A safety realignment framework via subspace-oriented model fusion for large language models
- Cultural Bias Matters: A Cross-Cultural Benchmark Dataset and Sentiment-Enriched Model for Understanding Multimodal Metaphors