Asking Twice, With and Without the Image: Phrasing Invariance as the Evidence in Multimodal Debate

Vision-language models, which answer questions about images, often state things that are not in the picture. A common remedy wraps the model in a small team of language-model agents. They decompose its description into claims, query vision tools, and debate the conflicts. Two things undermine this. The team is only as good as how its questions are worded, and the debating agents never see the image; they argue over text summaries of it. This proposal replaces "ask one well-worded question and argue about the answer" with a cheap measurement. Each claim is checked by asking the model many rephrasings of the same verification question, once with the image and once without it. A claim the image actually supports should get the same answer however the question is worded, and that answer should differ from what the model guesses blind. A claim that flips with wording, or matches the blind guess, is resting on language habits, not pixels.

That measurement does three jobs. It decides which claims need expensive multi-agent debate at all. It gives the text-only debaters a numerical, non-negotiable record of how much each claim depends on the image. It also makes the whole pipeline less sensitive to how any single prompt is phrased, because no verdict hangs on one wording. The idea connects four literatures. Prompt-sensitivity research supplies the probe. The cross-modal gap is what the probe measures behaviorally. Debate is the resolution mechanism for the cases the probe cannot settle. Hallucination benchmarks are the test.

Core Claim

The closest existing system is the post-hoc multi-agent pipeline of Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System. It has a supervisor language model that turns a caption into follow-up questions and tool queries, a detector (GroundingDINO) and a visual question answering model, and two text-only debaters who resolve conflicts from a shared pool of evidence. Its authors report that the system is highly sensitive to how the supervisor's prompts are phrased, and that badly worded follow-ups cause new hallucinations. Its debaters touch the image only through strings. We claim that both weaknesses have a common fix that needs no training.

For a claim $c$, we generate $K$ paraphrases $q_1,\dots,q_K$ of its verification question. We record the model's answers with the image, $a_k^{\text{img}}$, and with visual input removed, $a_k^{\text{blind}}$. Two quantities follow. The first is invariance $I(c)$, the fraction of sighted answers that agree with their majority. The second is visual dependence $D(c)$, the fraction of paraphrases on which the sighted and blind answers differ. We score

$$ g(c) = I(c)\cdot D(c), $$

which is high only when the answer is stable across wordings and would not have been given without looking.

A low $g$ is not a verdict of "false." A true claim about a highly predictable object can have low $D$ because the model's prior happens to be right. Low $g$ means "not verified by the image," and it is exactly the condition under which we spend tool calls and debate. What is new is using phrasing variation as a measurement instrument inside the verification loop, rather than as a nuisance to be trained away. The same measurement then gates debate and constrains what debaters may argue.

Research Program

The first component is the probe. For each atomic claim extracted from a caption, a fixed template generator plus a paraphrasing model produce $K$ verification questions (we sweep $K$ from 2 to 16). We fix the answer format to yes/no or a short span, because Evaluating Object Hallucination in Large Vision-Language Models showed that binary probing is far more stable across templates than caption metrics (standard deviation 0.78 versus 3.22). That makes per-item flips meaningful, not noise.

We implement the blind condition in two ways. The simple way is to remove the image or replace it with a blank. The sharper way reuses the value-vector masking of Mitigating Hallucination for Large Vision Language Model by Inter-Modality Correlation Calibration Decoding (IMCCD). That method masks the values of highly attended visual tokens to expose the model's language-prior distribution, and it uses the result to subtract logits at decoding time. We borrow its masking step, but we use the masked run as a claim-level witness, not as a logit correction.

The second component is gated, asymmetric debate. Claims with $g(c)$ above a threshold are accepted without further calls. The rest go to a two-round debate whose roles follow the asymmetric-information design of Removal of Hallucination on Hallucination: Debate-Augmented RAG. There, a retrieval-grounded proponent faces a parametric challenger. Here, a Visual Advocate holds the sighted answers and the tool outputs, and a Prior Challenger holds the blind answers and argues that the claim is a co-occurrence guess. A separate judge decides.

The design responds directly to Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate. That work found sycophancy strongly predicts abandoning correct answers ($r = 0.902$). It found that a designated judge cut disagreement collapse from about 81% to 41% on CommonsenseQA, and that gains vanish after 2–3 rounds. We therefore cap debate at two rounds, use a centralized judge, and make the paraphrase ledger immutable. Agents may dispute what the measurements mean, but not what they are, which removes the easiest route to social capitulation.

The third component is an analysis of when the probe works. Cure or Poison? Embedding Instructions Visually Alters Hallucination in Vision-Language Models found a sharp split: Qwen2.5-VL benefits from instructions rendered into pixels, while CLIP-based LLaVA-1.5 and InstructBLIP collapse to a Yes-ratio near 0.99. We predict a parallel split. The probe should be most informative for models with a large modality gap and strong yes-bias, where blind and sighted answers coincide most often on hallucinated claims. We test this across the same three models.

Baselines

The naive reference is the base model's own output with no intervention, evaluated with the default POPE template. For context, the IMCCD paper reports LLaVA-1.5 at 83.29% accuracy on POPE MS-COCO Random, with CHAIR$_i$ of 55.6%.

The strongest single-model competitor is IMCCD, a training-free decoding method. It reports 88.77% POPE accuracy on LLaVA-1.5 MS-COCO Random, above VCD's 87.73%, and it cuts CHAIR$_i$ from 55.6% to 49.8% while running at 7.38 tokens/s (IMCCD). We run its released procedure on the same models and splits. Beating it means exceeding 88.77% on that split with our pipeline applied to plain decoding. We also report our pipeline stacked on IMCCD, since the two are complementary.

The strongest revisor baseline is LURE, a trained model that masks high-entropy, late-position tokens and rewrites them. It reports MiniGPT-4 CHAIR$_s$ falling from 26.8 to 19.7 (Analyzing and Mitigating Object Hallucination in Large Vision-Language Models). On the same captions, our result must go below 19.7.

The closest system baseline is the multi-agent pipeline of Yu et al., with the same detector, question answering tool, debater model and round cap. For this system we found no reported result on Lacuna on POPE, CHAIR or FAITHScore, so we reimplement it and report our own numbers. To be fair to it, we give its supervisor prompt the same tuning budget we spend on paraphrase templates: an equal number of candidate prompts selected on a held-out split.

Three ablations isolate the key ingredient. The first uses paraphrases but no blind run ($g = I$). The second uses a blind run with a single phrasing ($K=1$, $g = D$). The third runs ungated debate on every claim. All methods share the same images, backbone models and call budget accounting, and we report language-model calls per image alongside accuracy.

Why not just…?

Why not just take a majority vote over paraphrased questions and skip debate entirely? This is the most serious threat, and it may well capture most of the gain on POPE-style existence questions. The majority answer inherits the model's yes-bias whenever that bias is stable across wordings, which POPE suggests is common for LLaVA-style models. Only the blind contrast exposes a stable-but-prior-driven answer. The $g = I$ ablation is exactly this alternative. If it matches the full pipeline on POPE Adversarial and on CHAIR, we will report that the blind contrast and debate are unnecessary for object existence, and we will restrict the claim to attributes and relations.

Why not just trust the object detector? For object existence, a detector like GroundingDINO may beat any language-model reasoning, and we include "detector only" as a baseline. Detectors do not verify attributes, counts or relations. LURE's authors also note that relational errors resist simple revision. So the decisive comparison is on the non-existence claims in CHAIR captions and the A-OKVQA/GQA POPE variants. The debate's value lies in resolving detector-versus-model conflicts there.

Why not just make the supervisor robust by meta-finetuning, as Promise does for prompts? Prompt-Robust Vision-Language Models via Meta-Finetuning contracts phrasing sensitivity in CLIP-style classifiers. It gains 1.13 harmonic-mean points over MMRL at an 18–22% training overhead. It tunes soft prompts for discriminative classification, not generative agents, and it requires training for every new backbone. Our approach is training-free and makes every verdict average over wordings by construction. We test the robustness claim directly by measuring output variance under paraphrased agent prompts.

Why not just run IMCCD and stop? IMCCD corrects token distributions. It cannot coordinate tool evidence or adjudicate claim-level conflicts. Its input phrasing still sets the attention pattern before any masking happens. If our pipeline adds nothing on top of IMCCD, the stacked condition will show it.

Proposed Experiments

We evaluate on LLaVA-1.5, InstructBLIP and Qwen2.5-VL. The benchmarks are POPE (Random, Popular and Adversarial, on MS-COCO, A-OKVQA and GQA), CHAIR$_i$/CHAIR$_s$ on MS-COCO captions, and MME. FAITHScore is used for free-form faithfulness. Metrics are accuracy and F1, Yes-ratio, CHAIR, calls per image, and the debate collapse and negative agreement rates from the sycophancy study, measured in our debates.

There are four decisive checks. The first is detection quality. The area under the ROC curve of $g(c)$ for flagging hallucinated claims must beat token entropy (LURE's signal) and both single-factor ablations. If it does not, the core claim fails.

The second is gating efficiency. Gated debate should match ungated debate's CHAIR within noise while using a fraction of the calls; DRAG reports roughly 10 calls per query as a reference cost in text.

The third is pipeline phrasing robustness. We rewrite the supervisor and debater system prompts in 20 paraphrases and report the standard deviation of POPE accuracy and the per-item flip rate. The pipeline should show markedly lower variance than the Yu et al. reimplementation under the same perturbations. This is the direct test of the limitation those authors named.

The fourth is the architecture split. The gap between $g$ and $I$ alone in detection quality should be largest on LLaVA-1.5 and InstructBLIP, and smallest on Qwen2.5-VL. If instead the blind contrast helps uniformly, the modality-gap interpretation is wrong, and we would report the signal as a generic prior detector.