Whose Trade-off Is It? Separating Generator and Judge Effects in Measured Faithfulness–Coverage Frontiers

Detailed image descriptions from vision-language models (systems that read an image and write text about it) face a known tension. Asking a model to mention more of what is in the picture makes it invent things that are not there. Making it stick strictly to what it can verify makes it terse and generic. The field now measures this tension mostly with automated judges: a strong language model reads each description, pulls out the objects, attributes, and relations it mentions, and scores two quantities. Faithfulness is the share of mentioned items that are actually in the image. Coverage is the share of the image's contents that got mentioned. The well-known frontier result, where GPT-4V reaches 28.0% coverage at 54.62% faithfulness while Emu2 reaches 74.98% faithfulness at only 8.1% coverage, comes from exactly such a pipeline (VALOR-EVAL). That pipeline uses one fixed judge prompt and one fixed generation prompt.

We propose to find out how much of a measured frontier belongs to the model being evaluated and how much belongs to the instructions given to the judge. The study crosses wording changes in the instruction to the describing model with wording changes in the judge's rubric. It then adds a set of caption rewrites whose true faithfulness and coverage are known by construction, such as pure restyling, padding with no new content, inserting one hallucinated object, or deleting one true detail. Running these together lets us attribute variance in the measured frontier to its sources. It also lets us test directly whether a judge responds to content or to style. The expected result is a corrected frontier, plus a cheap invariance check that any judge-based captioning benchmark can run on itself.

Core Claim

The claim is that open-ended multimodal evaluation has two separate sensitivity problems that current work blends together. The first is genuine generator brittleness: rewording the request changes what the model actually says about the image. The second is judge brittleness: rewording the rubric, or merely changing a caption's style, changes the score without any change in content. Two text-only findings point in opposite directions. Flaw or Artifact? compares regex-style answer parsing with an LLM judge across 12 prompt templates. It finds that most apparent prompt sensitivity disappears under the judge: on ARC-Challenge, the standard deviation of Gemma accuracy across templates falls from 0.28 to 0.005, and Spearman rank correlation of model rankings across templates rises from 0.30 to 0.92. In contrast, Rethinking Hallucinations shows that models give conflicting individual answers across prompts even when their aggregate scores match, with over 50% inconsistency on Med-HALT.

In the multimodal case, POPE reports that caption-based hallucination metrics vary across templates with a standard deviation of 3.22, against 0.78 for yes/no polling. That study could not say whether the larger number reflects generator behavior or scorer behavior. We expect neither text-only conclusion to carry over unchanged. Open-ended descriptions vary in length and style far more than multiple-choice answers do. Multimodal judges are also known to favor longer responses and to hallucinate visual details in their own reasoning (MLLM-as-a-Judge). Both properties should leak judge artifacts specifically into the coverage axis.

Research Program

The study crosses four factors. The first is the describing model: open models spread along the reported frontier, including LLaVA-1.5, InternLM-XComposer2, and Emu2, plus one frontier API model where budget allows. The second is the generation instruction. We use roughly eight wordings that span neutral requests ("describe the image"), exhaustive requests ("list every visible detail"), and cautious requests ("describe only what you are sure of"). We also include a few wordings that differ only superficially, so that pure brittleness can be separated from deliberate steering. The third is the judge rubric. We use roughly eight rewordings of the VALOR-EVAL two-stage extract-then-match prompt, varying emphasis, the order of instructions, and whether chain-of-thought reasoning is requested. The fourth is the judge's evidence: the raw image, or the human reference annotations as text. This last factor matters because judges have been found to score more reliably from text descriptions than from pixels (MLLM-as-a-Judge).

For each image $i$, describing model $g$, generation prompt $p$, and judge rubric $r$, we record faithfulness and coverage. We fit a crossed random-effects model:

$$s_{gpri} = \mu + a_g + b_p + c_r + (ab)_{gp} + (ac)_{gr} + (bc)_{pr} + \epsilon_{gpri}.$$

Here $s$ is either metric, $\mu$ is the grand mean, $a_g$, $b_p$, and $c_r$ are the effects of the model, the generation prompt, and the rubric, the paired terms are their interactions, and $\epsilon_{gpri}$ is residual noise. The quantities of interest are three shares of variance. The generator share is $a$ together with $(ab)$. The judge share is $c$ together with $(ac)$ and $(bc)$. The most consequential term is $(ac)$: if it is large, the judge's rubric re-orders models, which means the published frontier itself depends on the rubric.

The ingredient that turns this from a variance survey into a test is a set of content-controlled caption rewrites. Starting from human-verified captions, we produce four kinds of edits, and each has a known correct effect on the scores:

Edit type Correct effect
Restyle or reorder sentences No change in either score
Pad with generic, true-but-trivial filler Little or no change in coverage, since the filler names no annotated scene elements
Insert one absent object that typically co-occurs with the scene (the associative-bias case VALOR-BENCH targets) Faithfulness drops by a known amount
Delete one annotated element Coverage drops by a known amount

Rewrites are produced by a separate language model and then checked by annotators. A judge passes an edit type when its score moves in the correct direction and by roughly the correct amount. The pass rates directly measure whether the judge responds to content or to style.

Baselines

The naive reference point is VALOR-EVAL's own protocol, run as published: the original GPT-4 two-stage prompt, one generation prompt, and one score per caption (VALOR-EVAL). Its reported numbers on VALOR-BENCH are the frontier figures quoted above. It is also the strongest existing open-vocabulary judge for this question: it reports 82.35% accuracy on GPT-4V captions, against 5.88% for CHAIR (the classic hallucination metric that only recognizes 80 fixed object categories). We rerun it with the same judge model, the same images, and the same human anchor set used for every other condition.

The second baseline is a rubric ensemble that averages scores over our eight judge rubrics. It follows Auto-Prompt Ensemble, which reports ECE 0.041 against 0.317 for verbalized confidence on Reward Bench. ECE, or expected calibration error, measures the gap between a judge's stated confidence and how often it is actually right. We also run an image-clustered weighted variant in the style of Multimodal Bayesian Prompt Ensembles, which reports a 36% ECE reduction on HPSv2 for text-to-image preference rating. No reported result was found on Lacuna for either method on caption faithfulness or coverage. The ensembles receive the same number of judge calls that our full protocol spends.

The third baseline is a judge-free reference. A sentence-level entailment model checks each caption sentence against the human annotations, in the spirit of the entailment-based fidelity reward used by MOCHa. It has no rubric to perturb, so it shows what remains once rubric sensitivity is removed by design.

The key ablation drops the content-controlled rewrites and keeps only the factorial variance study. This isolates whether the rewrites, not just the extra prompts, are what reveal style-driven judge errors.

A judge setup counts as beating these baselines only if it meets three conditions:

  1. It agrees better with human-counted faithfulness and coverage on the anchor set.
  2. It has a higher pass rate on the restyling and padding rewrites.
  3. It keeps model rankings stable across rubrics.

Why not just…?

Why not just use one strong judge, since that made prompt sensitivity vanish? That result comes from multiple-choice and reading-comprehension tasks with text-only models (Flaw or Artifact?). There, the judge only has to recognize which answer was chosen. Coverage scoring asks the judge to enumerate the scene elements a free-form paragraph mentions. That is exactly the setting where verbosity bias and hallucinated reasoning have been documented (MLLM-as-a-Judge). This alternative is a real possibility, and our design can confirm it. If the judge share of variance is small, the $(ac)$ term is near zero, and restyle and padding pass rates are high, then the single-judge conclusion extends to multimodal description, and we will report that as the finding.

Why not just average over many judge prompts? Averaging reduces random variation but does not remove a bias shared by every rubric. If all rubrics reward longer captions, the average does too. The padding rewrites test this directly: an ensemble that inflates coverage for content-free filler fails no matter how well calibrated it is. If the ensemble passes the rewrites and matches human counts, the practical recommendation becomes "ensemble the judge," and the contribution shrinks to the diagnosis. We state that outcome in advance.

Why not just use yes/no probing like POPE and avoid judging free text? Yes/no probing removes the coverage axis entirely. CAPability shows that models often answer targeted questions correctly about details they leave out of their own descriptions. Probing therefore cannot measure the side of the trade-off we care about. We nonetheless report POPE-style accuracy for each model as a sanity anchor on the faithfulness side.

Proposed Experiments

VALOR-BENCH, with 211 images and human ground truth for objects, attributes, and relations, is the primary testbed. A human-annotated subset of CAPability adds breadth across its 12 dimensions. Annotators count true and hallucinated items for about 400 captions sampled evenly across the factorial cells. This anchor set turns every judge score into a measurable error.

There are three decisive checks:

  1. Variance shares. We estimate the generator and judge shares of variance separately for faithfulness and for coverage. Our hypothesis, stated so it can fail, is that the judge share is materially larger for coverage than for faithfulness.
  2. Frontier stability. We recompute the faithfulness-coverage frontier under every rubric and report the Spearman rank correlation of model orderings across rubrics. The text-only reference value is 0.92 (Flaw or Artifact?). A clearly lower multimodal value would show that published frontiers depend on the rubric.
  3. Corrected frontier. We re-score all models after length-controlled restyling and ask whether the coverage lead of verbose models, such as GPT-4V's 28.0%, shrinks relative to terse models such as Emu2.

Finally, holding the judge fixed at its most invariant configuration, we measure how far generation-instruction wording alone moves each model along the frontier. This answers, for open-ended multimodal description, the question POPE's 3.22 standard deviation left open: how much of that variation belongs to the generator, and how much to the scorer.