Subtracting the Language Prior: Atomic Visual Claim Verification for Multimodal Judges

Systems that read images and write text, called vision-language models, often describe things that are not in the picture. Checking for these errors at scale now depends on another vision-language model acting as a grader. That grader has the same weakness. It "sees" a mouse next to a laptop because mice usually sit next to laptops. A natural fix is to break each candidate answer into small, separately checkable claims ("there is a mouse", "the mug is red") and ask the grader about each one. We argue this fix fails in a specific and predictable way. The claims a candidate is most likely to have invented are the plausible-sounding ones, and those are exactly the claims a grader tends to confirm without looking. The grader's errors are correlated with the candidate's errors.

We propose to verify each atomic claim with a score that removes two biases. The first is the grader's general tendency to say "yes". The second is what the grader would say about the claim without the image at all. A claim counts as supported only when the actual pixels move the grader's belief beyond what the wording alone predicts. The result is a training-free judge that can be audited claim by claim. It runs on unannotated images, and we expect it to be resistant to padding answers with plausible filler. Our central empirical claim can be tested directly. Naive claim-by-claim judges accept more false claims as those claims become more plausible from text alone, and the corrected score should remove that trend.

What exists and what is missing

Three lines of work meet here, and none of them closes the loop. In text, Ask, Don't Judge (BinEval) breaks grading rubrics into batteries of yes/no questions. It reports a 0.563 Spearman correlation with humans on SummEval against G-Eval's 0.514, and 0.620 on the factual-consistency benchmark QAGS. This motivates decomposition, but it contains no vision. It also raises no concern that the answering model might be biased toward "yes". For multimodal judges, MLLM-as-a-Judge documents several failures: graders cluster scores high, prefer longer answers, and prefer their own style. It also found that "analyze-then-judge" reasoning hurts accuracy, because the grader invents visual defects in its free-form analysis. MR. Judge trains a 7B grader to reason before choosing between candidates and reaches 75.5% on VL-RewardBench. It still compares whole responses, though, and its reasoning is unconstrained text.

The only work that decomposes claims for multimodal hallucination is Pelican. It checks predicate-style sub-claims with generated code that queries object-detector outputs. It is a rewriting tool rather than a grader, and it is limited to the categories its detectors know. PerturboLLaVA's HalFscore extracts subject-relation-object triples with GPT-4o, but it needs dense human reference captions such as DCI.

The constraint that shapes our design comes from POPE. POPE showed that object hallucination follows co-occurrence statistics, and that models asked "Is there an X?" say "yes" up to nearly 99% of the time in some settings. POPE's MiniGPT-4 F1 falls from 78.86 with random negatives to 71.37 with co-occurrence ("adversarial") negatives, which is the prior at work. Put an atomic judge on top of a POPE-like verifier and you inherit this behavior. The circularity concern raised in Position: Adversarial ML for LLMs Is Not Making Any Progress becomes concrete here: judge and candidate share blind spots.

The obvious idea at this intersection is "BinEval for images", and it is not in the file. We treat it as our key ablation rather than our contribution. What is new is the diagnosis that atomic multimodal verification fails along the language prior, together with a correction that subtracts that prior.

Core Claim

Let $\ell(a, I)$ be the verifier's log-odds of answering "yes" to atomic claim $a$ given image $I$. Let $\bar a$ be a minimal counterfactual of the claim: the alternative attribute, the swapped relation, or the negated existence statement. Let $\varnothing$ denote a null visual input. The support score is a difference of differences:

$$s(a) = \big[\ell(a,I) - \ell(\bar a,I)\big] - \big[\ell(a,\varnothing) - \ell(\bar a,\varnothing)\big].$$

The first bracket compares the claim with its counterfactual on the same image. This cancels the generic tendency to answer "yes", because that tendency applies to both. The second bracket is the same comparison made without visual evidence, which measures how much the wording alone favors the claim. Subtracting it leaves the evidence contributed by the pixels.

A response's precision is the fraction of its atoms with $s(a)$ above a threshold. The threshold is calibrated once on a held-out POPE split. Pairwise preferences compare precision, with the count of supported atoms used to break ties. Using a ratio rather than a sum keeps padding from adding score automatically.

We claim three things. First, a naive atomic judge's false-accept rate rises with $\ell(a,\varnothing)$, and $s(a)$ flattens this trend. Second, this flattening is what drives better human agreement on hard hallucination cases. Third, because the decomposer never sees the image and the verifier never writes free-form analysis, the chain-of-thought failure from MLLM-as-a-Judge has nowhere to occur.

Research Program

The pipeline has three stages, and the design choice that matters is keeping them separate.

A text-only model splits the candidate response into self-contained atoms. It resolves pronouns into explicit referents and produces a counterfactual $\bar a$ for each atom. Because it has no image, it cannot add visual fabrications. We check decomposition fidelity by having a second model reconstruct the response from the atoms and measuring how much content is lost.

An open-weight vision-language verifier then scores each atom and its counterfactual, with and without the image. We use Qwen2.5-VL and LLaVA-1.5 because logits are available. Closed APIs get a sampling-based estimate.

Finally, scores are aggregated as described above. The cost is four short verifier calls per atom. The two null-image calls are text-conditioned and can be batched across candidates.

The analysis side is as important as the method. For each verified atom we record its prior plausibility $\ell(a,\varnothing)$, its claim type (existence, attribute, count, relation, text-in-image), and its gold label. From these we can draw the curve that tests the core claim: false-accept rate against prior plausibility, for each judge.

The choice of $\varnothing$ is an open design question. We will compare three options: a blank image, an unrelated image from the same domain, and the real image with the claimed region masked. The masked image is the most local but needs the claim's region, which we obtain from the verifier's own grounding output where it supports one.

The main risk deserves stating plainly. A claim that is both true and typical, such as a keyboard that really is next to the laptop, is penalized by the prior term. If the verifier perceives that object only weakly, the claim may fall below threshold. We therefore report recall on true atoms alongside precision on false ones, and we do not claim a free lunch.

Baselines

The naive reference points come first. The do-nothing predictor scores responses by length alone. It quantifies how much of any judge's agreement with humans could come from the verbosity bias documented in MLLM-as-a-Judge; a judge that does not clearly beat it has learned nothing about the image. The holistic judge is GPT-4o, plus the same open verifier we use, each giving a 1–5 score and a pairwise preference. Each is run with and without analyze-then-judge reasoning. The reported reference here is GPT-4V's Pearson similarity of 0.490 in scoring and roughly 79% human agreement in pair comparison on the MLLM-as-a-Judge benchmark.

The strongest existing method is MR. Judge. It is a trained reasoning grader that reports 75.5% overall accuracy and 83.2% on the hallucination subset of VL-RewardBench, against GPT-4o's 65.8%. It is the strongest competitor because it was trained specifically on synthesized hallucinations. If its weights are available we run it on our exact splits. Otherwise we compare on the published VL-RewardBench splits and state that the comparison uses reported numbers.

To be fair, our verifier uses the same 7B Qwen2.5-VL backbone, so the comparison is training-free correction against reinforcement-learning training at matched model size. Beating MR. Judge means exceeding 83.2% on the hallucination subset without any training. Matching it while also providing claim-level audit trails would be a weaker but still useful result.

The key ablation is the naive atomic judge: same decomposer, same verifier, raw "yes" answers, and precision aggregation as in BinEval. No reported result for a multimodal BinEval-style judge was found on Lacuna, so we establish this number ourselves. Two partial ablations isolate each bracket of $s(a)$: dropping the counterfactual term, and dropping the null-image term.

Two external variants complete the set. The first captions the image with the verifier and grades with text-only BinEval, which tests the description-beats-pixels finding of MLLM-as-a-Judge. The second is Pelican-style detector verification, restricted to the subset of atoms whose objects fall inside the detectors' vocabulary.

All judges get the same candidate responses, the same image resolution, and the same prompt-tuning budget, which is a fixed number of prompt variants selected on a shared development split.

Why not just…?

Why not just use a strong frontier model with a careful rubric and step-by-step reasoning? MLLM-as-a-Judge tested exactly this. Analyze-then-judge reasoning lowered accuracy because the grader invented visual details. Even GPT-4V's scoring correlation was only 0.490. On VL-RewardBench, GPT-4o trails a trained 7B grader by about ten points overall, according to MR. Judge. We include this judge anyway, because frontier models may have improved since then.

Why not just decompose and ask yes/no, as BinEval does in text? This is the real threat, and it is our main ablation. If the naive atomic judge's false-accept rate does not rise with prior plausibility, then the POPE-style "yes" bias does not carry over to verifying candidates' own claims, and our correction is unnecessary. The stratified curve settles this directly. If the curves for the naive and corrected judges coincide, we will report that atomic decomposition alone is enough.

Why not just caption the image and use a text judge? The judge then inherits every omission and fabrication in the caption. Fact-RLHF noted that even human COCO captions miss spatial and relational detail. Captioning is also weakest on charts and diagrams, which make up part of the MLLM-as-a-Judge benchmark. The caption-then-judge baseline, broken down by domain, will show where this approach holds up and where it breaks.

Why not just train a grader on synthesized hallucinations, as MR. Judge does? Training may learn to discount priors implicitly, so a trained grader could match us. Our method requires no training data, gives claim-level output, and can wrap any verifier, including MR. Judge itself. We therefore also run the correction with MR. Judge as the verifier. The two approaches are complementary if the corrected version improves further, and redundant if it does not.

Why not just verify with object detectors, as Pelican does? Detectors cannot check claims about text in images, chart values, or attributes outside their vocabulary. Pelican is also a rewriting pipeline, not a calibrated grader. On the in-vocabulary subset, the detector approach may be more accurate than ours, and we will report that subset separately.

Proposed Experiments

We need claim-level ground truth. We build it in two ways. First, we take responses from several vision-language models on DCI images and MMHal-Bench questions, and annotators label about 2,000 atoms against the image. These are stratified by claim type and by prior plausibility. Second, we build a controlled injection set using MR. Judge's reverse-synthesis idea. We insert false atoms into correct descriptions, drawn either at random or by POPE-style co-occurrence. This gives matched pairs that differ only in how plausible the inserted claim is from text alone.

The first decisive check is the false-accept-versus-prior-plausibility curve for the naive atomic judge, the holistic judges, and our score. We predict a rising curve for the naive judge and a flat one for ours, with the gap largest on co-occurrence insertions.

The second check is a padding attack. Correct responses are lengthened in two ways: with prior-plausible absent objects, and with neutral true filler. A judge resists verbosity bias if its score does not rise under either padding and falls under the first.

The third check is downstream agreement. We measure human correlation and pairwise agreement on the HQ and HARD splits of MLLM-as-a-Judge, and accuracy on the VL-RewardBench hallucination subset, against the reported 83.2%. We also re-rank the base LLaVA-1.5 model against Volcano and PerturboLLaVA outputs and compare the ranking with what MMHal-Bench and HalFscore produce. The reported figures for these are a Volcano MMHal score of 2.64 against 2.54 for the base model, and PerturboLLaVA's +2.4 HalFscore F-score. No absolute HalFscore value was found on Lacuna.

The proposal fails if the corrected score loses more than a few points of recall on true atoms while gaining little on false ones. It also fails if the trend with prior plausibility turns out to be absent in the naive judge to begin with.