Vision-language models, which pair an image encoder with a large language model, often describe things that are not in the picture: a knife beside a dining table, a refrigerator in any kitchen. An obvious fix is to have the model check its own answer, or to have several copies argue about it. The text-only literature says this mostly fails. A model asked to critique itself tends to turn correct answers into wrong ones. An LLM acting as verifier approved 84.45% of flawed plans in Can Large Language Models Really Improve by Self-critiquing Their Own Plans?. Debate also lost to plain majority voting at equal cost in Large Language Models Cannot Self-Correct Reasoning Yet. The common diagnosis is that the critic has no information the generator lacked. A vision-language model is in a different position, because the image is outside evidence that can be re-queried and physically edited. This proposal keeps the adversarial part of debate but moves the verdict away from any model's written judgment. Agents are assigned opposing stances about each visual claim. Each must point to where in the image its evidence lies. A claim wins only if the model's own belief in it drops when that region is masked out.
The idea in two sentences: a hallucinated object is supported by the language model's expectations, not by pixels, so hiding the pixels it supposedly rests on should barely change the model's confidence, while hiding a real object's pixels should destroy it. Debate with preset stances supplies explicit competing hypotheses ("it is a cutting board, not a knife") and the regions to test. A training-free masking test then settles the dispute, so no LLM judge and no external detector is needed.
Core Claim
We claim that the failure of intrinsic self-correction in vision-language models comes from using a verifier made of text, not from debate as such. We further claim that a region-masked visual-dependence test, applied to claims and counter-claims produced by stance-constrained agents, corrects object, attribute, and relation hallucinations in captions without training. We expect it to beat intrinsic self-correction, text-judged debate, and matched-compute voting, and to rarely delete true content.
For a claim $c$ (for example "there is a knife on the table"), an image $I$, a prompt $q$, and a region $R(c)$ that the defending agent's generation attended to most strongly, the visual dependence is
$$d(c) = \log p_\theta(c \mid I, q) - \log p_\theta(c \mid I_{\setminus R(c)}, q),$$
where $p_\theta$ is the same frozen model scoring the claim as text, and $I_{\setminus R(c)}$ is the image with that region replaced by a blurred or mean-colored patch. When a stance agent offers a counter-claim $c'$ about the same region, the judge keeps whichever of $c$ and $c'$ has the larger $d$. If neither exceeds a threshold calibrated on held-out data, the judge removes the mention.
Three pieces of prior work already sit at this intersection, and the proposal differs from each. CFMAD shows that assigning preset stances exposes hallucinations in text models, because agents defending falsehoods change position 10% more often. We reuse its stance mechanism but replace its LLM judge, since Chen et al. show that vision-language judges invent visual errors themselves. The post-hoc multi-agent system of Yu et al. and Woodpecker verify claims with Grounding DINO and BLIP-2. That inherits the tools' fixed vocabularies and caused a 6.2% mis-correction rate in Woodpecker; we use no external model. ViF passes visual tokens between collaborating agents through a learned block. We pass nothing between models and learn nothing.
Research Program
The work has three stages. The first is diagnostic. On COCO captions labeled by CHAIR, we measure how often a base model, asked "is X really in the image?", approves its own hallucinated objects. This gives a multimodal analogue of the 84.45% verifier false-positive rate. We also measure how often it rejects true objects, which is the correct-to-incorrect flip that Huang et al. report in text. If self-verification is already reliable here, the premise weakens and we say so.
The second stage builds the protocol. A base caption is parsed into object, attribute, and relation claims using the model itself. For each claim, one agent is assigned to defend it. A second agent is assigned to argue that it is wrong and must name the most plausible alternative at the same location, or state that the location is empty. The region $R$ comes from the model's own attention over image patches while generating the claim, so no boxes or detectors are involved. The known problem of attention landing on uninformative "blind" patches, identified in AVISC, is handled by also testing a randomly placed mask of equal size. The score we use is $d$ under the attended mask minus $d$ under the random mask. The surviving claims are then rewritten into the caption, with replacements where the counter-claim won. Replacing instead of only deleting is meant to avoid the brevity bias seen when models are tuned against hallucination.
The third stage is analysis. We break results down by hallucination type and by how strongly an object co-occurs with the scene, using the adversarial split logic from POPE. The prediction is that the gains concentrate on high co-occurrence objects, where language priors dominate. We run everything on LLaVA-1.5 and InstructBLIP, whose different connector designs matter for internal interventions according to RUDDER. All compute is reported in forward passes per caption.
Baselines
The naive reference is the base model's greedy caption with no intervention. Next to it is intrinsic self-correction: the same model critiques its caption against the image and revises it, with the prompt tuned for the same number of trials we spend tuning ours. This baseline carries the text literature's negative result into the visual setting, and we expect it to be at or below the naive caption.
Text-judged debate is CFMAD ported to images, with the same stance agents as ours and a vision-language judge that reads the transcript. It shares everything with our method except the verdict mechanism, so it is the most direct test of our claim. We must beat it on CHAIR$_i$ at equal passes. No reported result found on Lacuna for CFMAD on any multimodal benchmark; its text numbers (Hover 3-hop F1 0.6757 against 0.6476 for standard debate) motivate the port but are not comparable.
Matched-compute voting samples as many captions as our method's forward-pass budget allows. It keeps objects mentioned in a majority of samples. This is the baseline that beat debate on GSM8K, 88.2% against 83.0% at nine calls (Huang et al.).
The strongest single-pass training-free competitor is RUDDER. It reports a 23.6% relative reduction in CHAIR$_i$ and 24.4% in CHAIR$_s$ at 96% throughput (RUDDER). We also compare against OPERA, which reports about a 35% improvement over DoLa on CHAIR (OPERA). We rerun both with their published layer and strength settings, plus the same small tuning grid we give ourselves. To count as a win, we must reduce CHAIR$_i$ further than RUDDER on the same model while keeping object recall within one point. Otherwise we must show that the two methods are additive when stacked.
Woodpecker serves as a reference that is allowed external tools. It reports MiniGPT-4 POPE accuracy rising from 54.67% to 85.33%. Matching it without tools would be a strong result; falling short is acceptable if we beat it on relations, where its detectors are weak.
Two ablations isolate the key ingredients. The first scores claims by whole-image masking with no debate and no localization. The second keeps localization but drops the counter-claim agent, so there is only deletion and no replacement.
Why not just…?
Why not just take a majority vote over sampled captions? CFMAD argues that independent samples reproduce the same bias-driven hallucination, and co-occurrence hallucinations are exactly that case: every sample may mention the knife. The matched-compute voting baseline tests this directly. If voting equals us on the high co-occurrence subset, the debate adds nothing and we will report that.
Why not just check whether each claim depends on the image at all, without debate? This is the real threat. A whole-image contrast per claim resembles visual contrastive decoding turned into a verifier, and it may catch most existence hallucinations on its own. Our bet is that it fails on attributes and relations. There the image is relevant overall, but the specific claim ("red", "on top of") is not grounded in its region, and without a counter-claim there is nothing to replace it with. The first ablation settles this. If whole-image dependence matches us on attribute and relation claims, we drop the debate and publish the simpler verifier.
Why not just prompt better, or use single-pass steering? The Mirage of Performance Gains found that appending "Answer Yes if possible" reached 90.2% on POPE for LLaVA-13B, against 89.2% for VCD, so decoding tricks can look better than they are. For that reason we treat POPE as secondary, always report the "yes" rate, and anchor claims on generative CHAIR. RUDDER is far cheaper, so our method must either beat it clearly or combine with it. We run the stacked configuration to find out.
Proposed Experiments
The main table reports CHAIR$_i$, CHAIR$_s$, and object recall on COCO captions for both models, for every baseline and ablation, with forward passes per caption. Secondary evaluations are MMHal-Bench, where fine-tuned Volcano reports 2.64 against LLaVA-1.5-13B's 2.54 (Volcano), and the MME existence, count, position, and color subsets. POPE is reported on all three splits together with the "yes" rate.
Four checks decide the outcome. The first is the correct-to-incorrect rate: how often our method removes or replaces an object that is truly present. This must be below the intrinsic self-correction rate. The second is the gap between our method and text-judged debate at equal passes, which tests the core claim that the verdict mechanism, not the arguing, is what matters. The third is the gap between our method and whole-image dependence on attribute and relation claims, which tests whether debate earns its cost. The fourth is a mask-sanity control: replacing attended masks with random masks should erase most of the gain. If it does not, the method is exploiting a masking artifact rather than measuring grounding.
References
- Can Large Language Models Really Improve by Self-critiquing Their Own Plans?
- Large Language Models Cannot Self-Correct Reasoning Yet
- Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- Mitigating Large Vision-Language Model Hallucination at Post-hoc via Multi-agent System
- Woodpecker: hallucination correction for multimodal large language models
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
- Evaluating Object Hallucination in Large Vision-Language Models
- Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision-Language Models
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
- Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
- The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?
- Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revision