Prior-Breaking Demonstrations: Choosing In-Context Examples That Force a Vision-Language Model to Look

Vision-language models often describe objects that are not in the image. They say "yes, there is a mouse" because a laptop is visible, or they keep naming the same object down a list after the image has changed. The standard explanation is that the model's language habits override what its vision encoder sends in. Most training-free fixes act on the decoding step, and recent evidence suggests many of them mostly shift the model toward a particular answer rather than make it look harder. We propose a different training-free lever: the handful of worked examples placed in the prompt. We pick those examples by two criteria. First, the model can answer each one correctly only by using the image; its own image-free guess is wrong. Second, the examples are spread out in the space where the model actually receives visual information, the output of its projector. The bet is that a few such "prior-breaking" demonstrations teach the model, inside a single prompt, that its language habits are unreliable here, without fine-tuning and without the doubled per-token cost of activation steering.

Stated for a colleague in two sentences: score each candidate demonstration by how much the image changes the frozen model's own answer, then greedily choose a query-relevant set that is diverse in the model's projected visual-token space. If this reduces hallucination on benchmarks whose yes/no answers cannot be gamed, and raises the share of attention the model gives to image tokens, then exemplar selection is a real grounding tool rather than a way of nudging the answer distribution.

Core Claim

Two literatures are both partly right and each misses half the picture. The exemplar-selection literature shows that similarity-only retrieval produces redundant prompts. KITE uses information-theoretic, diversity-aware selection to beat nearest-neighbour and determinantal-point-process baselines on text classification. MSIER shows that image-similarity retrieval for multimodal prompts is redundant and "vision-biased," and that a retriever trained on the model's own utility does better. Neither measures hallucination. Both also optimize for task utility (accuracy, captioning score), and that objective can be met by examples the model would have answered from its priors anyway.

The hallucination literature shows the specific failure such examples should target. POPE traces "yes" answers to object co-occurrence statistics. HallusionBench shows models ignore edited images that contradict what they already believe. ROPE shows "hallucination by momentum": in multi-object sequences, attention to visual tokens falls below 20%. Our claim is that demonstrations selected for prior contradiction, and kept diverse in projected-feature space, target exactly this failure. Utility-selected or similarity-selected demonstrations do not.

The closest existing work is IC-VCO. It places an original image and an edited negative in a shared multi-image context, but only as a fine-tuning objective, using 19,453 synthesized edits. We keep the intuition that contrast inside the context matters and drop the training and the image editing. Our contrast comes from real images that the frozen model itself gets wrong without looking.

Research Program

Fix a frozen model that accepts several images in one prompt. Let $q$ be a question and $a$ its correct answer for image $x$. For each candidate demonstration $e=(x,q,a)$ in a labeled pool, compute its prior-contradiction gain

$$g(e) = \log p_\theta(a \mid x, q) - \log p_\theta(a \mid q),$$

where $p_\theta$ is the frozen model's probability of the answer tokens with and without the image. A large $g$ means the image changes the model's answer toward the truth. This is the same image-versus-no-image contrast that TLVS computes at every decoding step. The difference is that we compute it once per pool item, offline. Per-query inference then pays only for a longer prompt, not for a second forward pass at every token. Because $g$ is defined on whole answer sequences, it applies to captions as well as yes/no answers. This sidesteps the restriction to single-label outputs that KITE acknowledges.

For a test query $Q$, we select $k$ demonstrations greedily, in the style of maximal marginal relevance (repeatedly add the item that is relevant but unlike what is already chosen):

$$e^* = \arg\max_{e \notin S}\; \lambda\,\mathrm{rel}(e,Q) + \beta\, g(e) - (1-\lambda)\max_{e' \in S}\mathrm{sim}_{\pi}(e,e').$$

Here $S$ is the current set, and $\mathrm{rel}$ is query relevance computed from concatenated CLIP image-and-text embeddings, as in the unsupervised MUIER baseline of MSIER. The term $\mathrm{sim}_{\pi}$ is cosine similarity between mean-pooled outputs of the model's own projector, the layer that maps vision-encoder features into the language model's input space. Measuring diversity there, rather than in CLIP space, is a deliberate choice. MSIER found that the visual side of the retrieval space had to be reshaped toward the model's preferences. Deep Pre-Alignment argues that what the language model receives after a lightweight projector is geometrically different from the encoder's own space. We test this choice directly rather than assume it.

The program has three parts. First, we build the scoring and selection pipeline on COCO-train-derived question, answer, and caption pools. These pools are kept disjoint from the evaluation images, since POPE and CHAIR draw on COCO val. Second, we evaluate on discriminative and open-ended hallucination benchmarks. Third, we probe the mechanism: whether the fraction of attention on visual tokens at object-naming positions rises under our prompts. ROPE's measurement of attention below 20% gives a concrete reference point. We cap $k$ at 8. The finding in many-shot test-time adaptation that long demonstration lists distract the model and cost quadratic attention constrains the design to few, well-chosen examples.

The models are LLaVA-NeXT-Interleave, which IC-VCO also uses and which handles multi-image prompts, and Qwen2.5-VL, which TLVS evaluates. Single-image models such as LLaVA-1.5 handle interleaved demonstrations poorly. We therefore report them only as a stress case, not as a primary result.

Baselines

The naive reference point is the zero-shot model with its default prompt. Next to it is random selection of $k$ demonstrations from the same pool. HW-TSC reports that random selection can beat similarity retrieval because it is more varied. The similarity baseline is image-only nearest-neighbour retrieval (RICES), alongside the unsupervised image-plus-text variant MUIER. On COCO captioning, MSIER reports CIDEr 99.22 for RICES and 100.65 for MUIER. The strongest exemplar-selection competitor is MSIER itself, which reaches CIDEr 106.17 at 4 shots. We retrain it on our pool with utility labels from the same frozen model and the same number of scoring forward passes we spend computing $g$, so supervision cost is matched.

The strongest training-free hallucination competitor is TLVS. TLVS reports that on LLaVA it lowers CHAIR_s from 46.0 to 31.2 and raises average POPE accuracy from 85.09% to 89.78%. We rerun it on our models with its released settings and report wall-clock latency alongside accuracy. We also include VCD and the prompt-only adjustment from the contrastive-decoding critique. That paper reports the prompt adjustment at 90.2% versus VCD at 89.2% POPE accuracy on LLaVA-13B (COCO Random), which shows how easily POPE is moved by answer bias alone. As an upper reference that uses training, we cite IC-VCO's reported gain of 1.24 points on LLaVA-NeXT-Interleave. We do not claim to beat a fine-tuned model.

Three ablations isolate the key ingredients. Gain-only drops the diversity term. Diversity-only drops $g$. CLIP-space diversity replaces $\mathrm{sim}_\pi$ with CLIP similarity. A fourth control, label-balanced nearest-neighbour, forces half of the chosen demonstrations to have "no" answers. It separates real grounding from simple correction of the answer prior.

Beating the field means two things on the same models and pools. On CHAIR_s and ROPE, the method must reduce hallucination more than the best similarity, random, or MSIER prompt at equal $k$. It must also match or exceed TLVS's relative CHAIR_s reduction at lower latency. And it must do so while the label-balanced control and the prompt adjustment do not.

Why not just…?

Why not just use random or nearest-neighbour exemplars? Random and similarity selection are cheap, and random selection already adds variety. However, MSIER found 4 well-chosen examples beat 32 random ones. Neither random nor similarity selection prefers examples the model gets wrong without looking. Our diversity-only and random arms test whether variety alone is enough. If they match the full method on ROPE and CHAIR, the prior-contradiction score is unnecessary and we will say so.

Why not just balance the labels or tell the model to be careful? This is the most serious threat. The contrastive-decoding critique shows that a one-line instruction moves POPE further than VCD does, so any demonstration set with more "no" answers might look like a hallucination fix. We address this in three ways. We include the label-balanced and prompt-adjustment controls. We report the model's yes-ratio next to every POPE number. And we treat open-ended CHAIR, ROPE's sequential probes, and HallusionBench's control pairs as the primary evidence, because on these an answer-bias shift cannot produce gains. If balanced prompts match us there, the idea fails.

Why not just use contrastive decoding or token-level steering? Contrastive decoding gave no gain on CHAIR, NoCaps, or MME in the critique. TLVS does work, but it doubles decoding cost and, by its own account, cannot recover objects the projector never encoded. Our method shares TLVS's limit on encoding. Its advantage is cost: the image/no-image contrast is paid once per pool item rather than at every token. The latency-matched comparison decides whether that cheaper route is competitive. The two methods could also be combined, and we test the combination.

Why not just fine-tune with in-context contrast, as IC-VCO does? If one can train, IC-VCO is a reasonable choice. Our target setting is a frozen or closed model, or a deployment that cannot pay for synthesizing edited images. Its acknowledged bias toward existence and attribute edits also suggests that real prior-breaking examples may cover relational errors better. We test this on AMBER's relation split.

Proposed Experiments

The primary experiment runs every arm on POPE (all three negative-sampling settings, with yes-ratio reported), on CHAIR_s and CHAIR_i for open-ended captioning, on ROPE under default, student-forcing, and teacher-forcing prompting, on HallusionBench question-pair accuracy, and on AMBER, for $k \in \{2,4,8\}$ and both models. To check general task utility, we also evaluate on COCO captioning (CIDEr) and OK-VQA, against MSIER's reported numbers.

The decisive checks are the following. First, the ROPE teacher-forcing condition, where each item ends in a different object after a run of identical ones and the final-item accuracy is reported to fall to near zero. Momentum-breaking demonstrations should help most there. Second, attention analysis showing a higher share of attention on visual tokens at object-naming positions under our prompts than under nearest-neighbour prompts. If accuracy rises but attention does not, we will report the gain as a prompt-format effect rather than grounding. Third, the comparison of projector-space and CLIP-space diversity. If the two are indistinguishable, the cross-modal-projection part of the claim is dropped and the contribution reduces to selection by prior contradiction.