Vision-language models (systems that read an image and answer questions about it in text) often describe objects that are not in the picture. A popular fix wraps the model in a team of text-only language-model "agents." One agent decides which claims look suspicious, external tools such as an object detector check them, and other agents argue about the result before a final answer is written. These pipelines raise average benchmark accuracy. Each agent, however, is driven by a hand-written instruction, and language models are known to change their answers when such instructions are reworded trivially. We propose to measure, instance by instance, how much a multi-agent correction pipeline's final answers flip when its agents' instructions are reworded. We then locate which agent carries most of that instability and stabilize only that agent by giving it a few worked, text-only examples of verification, instead of a longer or better-tuned instruction.
The bet is simple to state. Multi-agent correction removes some hallucinations, but it may turn the remaining errors into prompt-dependent coin flips that average accuracy hides. A handful of demonstrations given to the single agent that decides what to verify and how to read tool outputs should convert many of those coin flips into consistently correct answers. It should do so more cheaply than voting over many prompt variants.
Core Claim
The authors of the most direct precursor, Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System, built a pipeline in which an LLM Supervisor reviews a vision model's description, sends follow-up questions to a detector (GroundingDINO) and a question-answering model, and has two debater agents reconcile the evidence. They report that this pipeline is highly sensitive to the Supervisor's prompt phrasing, and that poorly phrased questions can trigger new hallucinations. No later multi-agent work in the file takes this up. Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow intervenes on attention inside the model. InEx adds introspection and image-editing agents with static prompts. LEDGERMIND replaces free-text history with a structured evidence ledger. All of them report aggregate accuracy and use zero-shot instructions for every agent.
Our claim has three parts, each testable:
- Under semantically irrelevant rewordings of agent instructions, a multi-agent correction pipeline shows substantial instance-level answer instability, even where it improves aggregate accuracy.
- That instability is concentrated in the agent that turns a description into verification queries and turns tool outputs into verdicts, which we call the verifier role.
- Anchoring that role with a few text-only worked verification traces reduces instability more per unit of compute than paraphrase voting or prompt optimization, and it does so by producing consistent correct answers rather than consistent wrong ones.
Why This Combination, and What It Borrows
The measurement comes from Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity. For text-only models, that paper defines ambiguity as the share of questions that receive conflicting answers across prompt variants. It splits behavior into consistently correct, consistently wrong, and prompt-dependent randomness. It finds ambiguity often above 50%, and it shows that retrieval augmentation lowers hallucination while making errors more prompt-dependent. That retrieval finding is the strongest motivation here: tool-calling correction pipelines are structurally similar to retrieval, so we expect the same error-shifting. The paper never examines vision models or agent pipelines, and we carry its metrics over to both.
The intervention comes from few-shot stabilization work, which constrains how we can apply it:
- ProSA reports that moving from zero to one demonstration gives the largest robustness gain across its prompt templates.
- Robustness of Learning from Task Instructions shows that demonstrations let a model recover the task even under irrelevant instructions.
- What Did I Do Wrong? finds that a single example per class beats elaborate instructions for consistency.
- POSIX argues that instruction tuning itself induces hypersensitivity to phrasing.
Two findings constrain where demonstrations can go. Why LVLMs Are More Prone to Hallucinations in Longer Responses shows that vision models shift attention from image tokens to their own text as context grows. ViF measures visual attention dropping by up to 60% by the twentieth agent turn. Stuffing image demonstrations into the vision model's context is therefore likely to hurt. For this reason we place demonstrations only in the text-only verifier agent, whose job is to reason over claims and tool outputs, never over pixels. The vision model's context stays untouched.
Demonstrations also bring their own instability. On the Relation between Sensitivity and Accuracy in In-Context Learning shows that example order and selection change predictions. It also shows that prediction instability under perturbation predicts error, which gives its SENSEL selective-prediction method a +4.1 point gain in area under the F1–coverage curve on GPT-J-6B over probability-based scores. We reuse that idea in two ways. First, we test whether anchoring merely trades instruction sensitivity for example sensitivity. Second, we use disagreement across perturbed verifier prompts as a signal for when to call tools.
Research Program
We first reimplement a Yu-style pipeline with the components they describe:
- a vision model producing an initial answer or description (LLaVA-1.5-7B and Qwen-VL, matching InEx's open models);
- a verifier agent;
- GroundingDINO plus a question-answering model as tools;
- two debater agents and a final writer.
For the text agents we run an open instruction-tuned model, Llama 3-70B-Instruct, which ProSA flags as unusually stable, and a frontier API model, GPT-4o, which ProSA flags as an "unstable genius." The contrast tells us whether the effect depends on the orchestrator's baseline stability.
We build a perturbation set for every agent's instruction from documented sources of brittleness. These are:
- label and format changes, as in DOVE;
- trailing spaces and appended courtesies, as in The Butterfly Effect of Altering Prompts;
- ProSA's role-player, emotional, and output-requirement template families;
- meaning-preserving paraphrases checked by two annotators.
For an instance $i$ and a set $P$ of pipeline prompt configurations, let $a_i(p)$ be the final answer under configuration $p$. Ambiguity is the fraction of instances whose answers are not all identical:
$$A = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[\,\exists\, p, p' \in P : a_i(p) \neq a_i(p')\,\right],$$
where $N$ is the number of evaluation instances. We also report self-consistency, the probability that two configurations drawn at random agree, so that the numbers compare directly with the text-only study.
To locate instability, we compute a per-role ambiguity $A_r$. This perturbs only agent $r$'s instruction while holding the others at their default text, and it is a one-at-a-time attribution. To check that roles do not interact strongly, we also run a small joint-perturbation grid.
The intervention is a verifier prompt that keeps the original instruction and appends three or four worked traces. Each trace shows a description, the claims extracted from it, the tool queries issued, the raw tool outputs, and a verdict with a one-line justification that cites the tool output.
The traces come from images disjoint from every evaluation split, such as COCO training images when evaluating POPE's validation images. They deliberately include one case where the detector contradicts the vision model and the verifier sides with the detector. They also include one case where the detector is inconclusive or plainly wrong and the verifier abstains or defers to the question-answering model. Without that second case, anchoring would teach the verifier to obey the detector blindly.
Finally, a cost-control variant runs the verifier under two or three perturbed instructions. It escalates to full debate only when they disagree, following the SENSEL logic.
Baselines
The naive reference is the vision model alone, under the same prompt-perturbation protocol applied to its question template. It shows how much instability exists before any agents are added. The second reference is the zero-shot multi-agent pipeline, our reimplementation of Yu et al.. The file contains no reported POPE or FAITHScore number for this system, so our reimplementation's numbers are the reference.
The strongest published competitor on aggregate accuracy is InEx. It reports improving LLaVA-1.5's POPE accuracy by 8.9% on the MSCOCO subset and an MME-Hall score of 673.3, outperforming the decoding methods OPERA, VCD, and ICD. We run InEx's published configuration on the same models and splits and subject its agent prompts to the same perturbation protocol. This makes it both an accuracy competitor and a second pipeline whose multiplicity we measure. For HallusionBench, the reference point is the benchmark paper's own result: HallusionBench reports 31.42% question-pair accuracy for GPT-4V, with all other evaluated models below 16%.
Two compute-matched baselines test the specific claim. Paraphrase voting runs the zero-shot verifier under $K$ sampled rewordings and takes a majority. We set $K$ so that its total verifier tokens equal those of the anchored verifier, whose prompt is longer. Optimized prompt selection searches the perturbation space on a development split for the single verifier instruction with the best accuracy, with the same search budget we spend choosing exemplars.
Three ablations isolate the key ingredient:
- anchoring all agents rather than only the verifier, which tests whether localization matters;
- anchoring with demonstrations of correct verdicts but no tool-disagreement or tool-failure cases, which tests whether the content of the traces matters or just their presence;
- replacing the traces with an equally long instruction that describes the same procedure in prose, which tests examples against more instruction text.
Beating the competitors means three things together. Ambiguity must be lower than both compute-matched baselines on HallusionBench and on POPE's adversarial split, under every orchestrator model. Aggregate accuracy must be no worse than the zero-shot pipeline. And the instances that leave the "random" category must land mostly in "consistently correct," not "consistently wrong."
Why not just…?
Why not just vote over many prompt rewordings? Majority voting across paraphrases is the most direct way to cut answer variance, and it is a real threat. It needs no exemplar design and is easy to parallelize. Two things limit it. Its cost scales with $K$ for every instance. And averaging cannot repair an error that most rewordings share; voting moves "random" instances toward whatever the majority believes, which may be the language prior. The compute-matched comparison settles this directly. If voting matches anchoring in both ambiguity and correct-consistent share at equal tokens, we will report that the cheaper intervention suffices and that demonstrations add nothing. The gated variant also tests whether a little voting, used only as a disagreement detector, can be combined with anchoring.
Why not just find the best supervisor prompt? The evidence in the file says optimized prompts stay fragile. Trailing spaces flip classifications in The Butterfly Effect of Altering Prompts. Label changes swing leaderboard ranks by up to eight positions in When Benchmarks are Targets. A prompt chosen for development accuracy is not chosen for stability under rewording. Our optimized-prompt baseline is evaluated under the full perturbation set, so if it turns out stable, the experiment will show it.
Why not just call the object detector and skip the agents? On POPE this may genuinely win. Its questions are pure object-existence probes, which is exactly what GroundingDINO answers, and the vision models' near-99% "yes" rate reported in Evaluating Object Hallucination in Large Vision-Language Models makes any calibrated detector look strong. We will report detector-only POPE results and not claim POPE as the decisive testbed. HallusionBench's edited charts, maps, and figures, and open-ended descriptions scored for factual alignment, involve attributes and relations a detector cannot settle. There, interpreting tool output is the hard step, and that step is what anchoring targets.
Why not just intervene inside the model, as ViF or representation steering do? ViF reports cutting snowballing by at least 30% with 8–13% added latency. SHARP steers hidden activations. Both need access to the weights and act on visual attention. Neither addresses which claims get checked or how tool verdicts are read, and both are compatible with our change. We include one ViF-equipped vision model as a combination condition to test whether the gains add up.
Decisive Experiments
The study runs on POPE's random, popular, and adversarial splits; HallusionBench, using its control pairs and its decision tree that separates language hallucination (prior overrides image) from visual illusion (misread image); and CHAIR for open captions. The POPE authors report CHAIR fluctuating with a standard deviation of 3.22 across prompts against 0.78 for POPE. That makes CHAIR a natural stress test for prompt multiplicity, and we will report per-template distributions rather than single scores.
The first check is the premise. If the zero-shot pipeline's ambiguity is no higher than the base vision model's, then multi-agent correction does not create hidden instability, and the main motivation fails. We report that negative result as the finding.
The second check is localization. If per-role ambiguity is spread evenly across agents rather than concentrated in the verifier, single-role anchoring loses its rationale, and the all-agent ablation becomes the main condition.
The third check is the head-to-head against paraphrase voting and prompt optimization at matched compute, as defined above.
The fourth check guards against stabilizing the wrong thing. We resample and reorder the exemplar set five times, following the sensitivity–accuracy study. If ambiguity across exemplar sets exceeds the instruction-induced ambiguity it removed, anchoring has only moved the brittleness. We also track HallusionBench's split between language hallucination and visual illusion. Anchoring a text verifier should mainly reduce language-hallucination errors, and if it instead reduces visual-illusion errors, our account of why it works is wrong.
The last check tests the SENSEL-style gate. Agreement across perturbed verifier prompts should predict final correctness better than the vision model's maximum token probability, measured as area under the accuracy–coverage curve. If it does, the gate can skip debate on agreeing instances, and we report the fraction of instances skipped at equal accuracy.
References
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- InEx: Hallucination Mitigation via Introspection and Cross-Modal Multi-Agent Collaboration
- LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
- Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity
- ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs
- Robustness of Learning from Task Instructions
- What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineering
- POSIX: A Prompt Sensitivity Index For Large Language Models
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- On the Relation between Sensitivity and Accuracy in In-Context Learning
- DOVE: A Large-Scale Multi-Dimensional Predictions Dataset Towards Meaningful LLM Evaluation
- The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models
- When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
- Evaluating Object Hallucination in Large Vision-Language Models
- SHARP: Steering Hallucination in LVLMs via Representation Engineering