Prompt Search Against Judges Who Share the Model's Blind Spots

Large vision-language models, which pair an image encoder with a text generator, often describe objects that are not in the picture. They do this most when an absent object would be likely given the scene: a toaster on a kitchen counter, a ball beside a dog. A simple handwritten instruction that tells the model to trust the image over its expectations already reduces this. The natural next step is to search over instructions automatically. Automatic search needs a score, though, and for open-ended descriptions the only scalable scorers are other multimodal models acting as judges. Those judges are trained on the same kinds of data and carry the same expectations. A judge that thinks toasters belong on counters will not flag an invented toaster. A prompt search driven by such judges will therefore cut the hallucinations judges can see and leave alone, or even reward, the ones they share with the model.

This note proposes to treat a panel of model judges as noisy annotators whose error rates depend on how plausible the claimed object is given the scene. Each judge's reliability is estimated with a classic latent-label model. That model is anchored on the subset of claims where ground truth is free, namely objects covered by COCO annotations. The resulting posterior probability that each claim is grounded becomes the reward for automated prompt search. The bet is that correcting for shared, plausibility-dependent judge bias is what separates a prompt that reduces hallucination in human eyes from one that only reduces it in judge eyes.

Core Claim

Probabilistic label aggregation is the Dawid–Skene approach of Smyth et al. (1994) and Snow et al. (2008). It infers hidden true labels together with a per-annotator "confusion matrix," meaning the probability of each response given each true label. It assumes annotators err independently once the true label is known. STABLEVAL names this assumption as its main failure point when annotators share a bias.

Model judges of hallucination violate the assumption in a predictable direction. Their errors cluster on scene-plausible absent objects, which are exactly the adversarial negatives of POPE. We claim that making each judge's confusion matrix depend on a plausibility stratum, and fixing those matrices with anchor claims of known truth, gives a reward that tracks human judgment much better than majority vote or a single strong judge. We further claim that prompts optimized against this reward lower human-verified hallucination on held-out images more than prompts optimized against any cheaper reward. The second claim is the one that matters, and it can fail. If a judge-free reward such as CHAIR yields equally good prompts, the aggregation machinery is not worth its cost.

Research Program

The target task is free-form image description by LLaVA-1.5-7B and 13B. We chose these models because the strongest training-free comparators report numbers on them. We add one black-box API model to test the setting where decoding interventions are unavailable.

Each generated description is split into atomic claims about objects, attributes and relations by a text-only LLM. Each claim $c$ is assigned a plausibility stratum $s_c$, computed by binning the co-occurrence of the claimed object with the objects actually present, using caption-corpus statistics.

A panel of $J$ model judges, mixing open and API models in direct-verification mode, labels each claim as grounded or hallucinated. We avoid chain-of-thought mode, because MLLM-as-a-Judge found that judges invent details in their intermediate reasoning. The aggregation model is

$$ P(y_{jc} = b \mid z_c = a, s_c) = \pi_j^{(s_c)}[a, b], $$

where $z_c$ is the latent truth of claim $c$, $y_{jc}$ is judge $j$'s verdict, and $\pi_j^{(s)}$ is judge $j$'s 2×2 confusion matrix within stratum $s$. Fitting uses expectation–maximization (EM). For claims about the 80 COCO categories, $z_c$ is clamped to its known value rather than inferred. These anchors are what identify the stratum-specific matrices even when every judge errs in the same direction. Without anchors, a correlated bias cannot be told apart from truth.

The reward for a prompt $p$ on images $x$ is the posterior expectation

$$ R(p) = \mathbb{E}_x \sum_{c \in C(p,x)} \big[\, q_c - \lambda (1 - q_c) \,\big], $$

where $C(p,x)$ is the claim set produced under prompt $p$, $q_c$ is the posterior probability that claim $c$ is grounded, and $\lambda$ sets the price of a hallucination relative to a correct detail. Counting correct claims is deliberate. Any pure hallucination-rate reward is maximized by prompts that make the model say almost nothing.

The search procedure follows Error Taxonomy-Guided Prompt Optimization, which builds a taxonomy of failure categories from many runs and writes guidance targeting the most frequent ones. Here each failure is weighted by its posterior hallucination probability, so claims the panel disagrees on because of judge noise count less. Following UniAPO, we store failures as text rather than re-feeding images to the optimizer, which avoids exhausting its context. Search starts from the Magnifier Prompt instruction, so any gain is measured as improvement over the best known handwritten prompt.

Unlike STABLEVAL and RE-PO, which use EM over text-only human panels for ranking and weight training respectively, here aggregation drives a discrete prompt search over images. The judges are models with a known, shared failure direction rather than independent crowdworkers.

Baselines

The naive reference point is the target model's default prompt with no intervention. A slightly stronger version of it is the original Magnifier Prompt, the strongest handwritten instruction in the file. It reports POPE Adversarial macro F1 of 84.5 on LLaVA-1.5-13B, against 81.9 for visual contrastive decoding. Its paper reports no long-form claim-level hallucination rate, so we rerun it under our human evaluation.

The strongest non-prompt competitor is RITUAL. It decodes with both the original and a randomly transformed image, and it raises LLaVA-1.5 POPE accuracy from 84.13% to 88.87%. It is the strongest because it needs no labels or judges at all. We run its released settings on the same images and report its doubled inference cost alongside its accuracy. We have no reported CHAIR number for RITUAL (none found on Lacuna for RITUAL on CHAIR), so we measure it ourselves.

The strongest prompt-search competitors share our optimizer, image pool, number of target-model calls and number of optimizer tokens, and differ only in the reward. The first uses majority vote over the same judge panel. The second uses the single best judge alone. For context on that judge, MLLM-as-a-Judge reports GPT-4V scoring correlation with humans of 0.490 and roughly 79% pairwise agreement. The third uses plain EM aggregation without plausibility strata or anchors. The fourth uses CHAIR alone, computed from COCO annotations.

The plain-EM variant is the ablation that isolates the key ingredient, the modeling of shared plausibility-dependent bias. A second ablation keeps the strata but drops the anchors.

Beating the baselines means two things. First, the proposed reward must yield a lower human-verified hallucination rate on held-out images than every other reward, at an equal or higher count of correct claims. Second, the gap must be concentrated in the high-plausibility stratum. If the gap sits elsewhere, the stated mechanism is wrong even if the numbers look good.

Why not just…?

Why not just take a majority vote of the judges? Majority vote cannot fix errors the majority shares, and that is precisely the failure we expect on plausible absent objects. Evidence from text panels shows vote-based aggregation is fragile. STABLEVAL reduces MT-Bench rank standard deviation from 0.259 to 0.197. Deep Learning from Crowds reports 76.74% versus 83.15% on LabelMe. Neither result concerns correlated model judges, so the majority-vote reward arm tests this directly. If majority vote matches us in the high-plausibility stratum, the claim fails.

Why not just optimize prompts against CHAIR, which needs no judges? This is the most serious threat. CHAIR is exact for the 80 COCO categories and costs nothing. Its weaknesses are coverage and exploitability: it ignores attributes, relations and objects outside those categories, and a search can satisfy it by steering descriptions away from COCO nouns. Our anchors use the same COCO information, so the honest comparison is whether judge-scored claims outside COCO's coverage add anything. The held-out human evaluation scores all claim types. We will report it separately for COCO-covered and uncovered claims. If CHAIR-optimized prompts match ours on uncovered claims, we will recommend CHAIR.

Why not just optimize on POPE, where labels are clean? POPE asks yes/no existence questions. A prompt that fixes yes-bias on those questions need not change what the model volunteers in a long description, and long outputs drift away from the image as they grow, as shown in Why LVLMs Are More Prone to Hallucinations in Longer Responses. We include a POPE-optimized prompt as an extra reward arm and test whether it transfers to long-form generation.

Why not just use decoding fixes like RITUAL? RITUAL needs no supervision, but it needs a second forward pass per query and, like contrastive decoding, cannot run on API-only models. A discovered prompt adds no inference cost. On open models we report both methods and their combination, because the two may stack.

Proposed Experiments

Optimization uses 500 COCO train images and the judge panel. Evaluation uses 300 held-out COCO val images plus a subset of HallusionBench images, whose edited controls probe whether prompts help when the image contradicts expectations. Three human annotators label every claim on the evaluation images. Their labels are themselves aggregated with Dawid–Skene, and inter-annotator agreement is reported.

The primary metrics are the claim-level hallucination rate and the number of correct claims per image, broken out by plausibility stratum. Secondary metrics are CHAIR, POPE Adversarial macro F1 and HallusionBench question-pair accuracy, which test whether a prompt tuned for descriptions harms or helps discriminative probes.

A separate check measures the reward itself. For each aggregation scheme, we report the agreement of its claim posteriors with human labels per stratum, before any prompt search. This is the first decisive check: if stratified, anchored EM does not beat plain EM on high-plausibility claims, the mechanism is absent and downstream gains would be coincidence.

The second decisive check is the reward-hacking audit. For every reward arm, we compare the judge-estimated gain with the human-verified gain on held-out images. The proposal predicts that this gap is largest for majority vote and single-judge rewards, and smallest for ours.

The third decisive check tests transfer. A prompt found on LLaVA-1.5-13B is applied unchanged to the black-box model, to see whether the gain survives where decoding methods cannot be used.