Competitive Linking Between Checklist Criteria and Response Sentences for LLM Judges

Many automatic evaluators for chatbots now work from a checklist. A strong model writes five to ten yes/no questions for each user prompt, and a cheap model answers each question about a candidate response. The score is the fraction of "yes" answers. This is cheap and agrees well with human preferences. It has two known weaknesses. First, nothing stops the checklist from containing near-duplicate questions, which count the same evidence twice. Second, nothing penalizes a response for content that no question asks about, so a response can be padded with irrelevant text and keep its score.

Statistical machine translation faced the same structure twenty years ago. It extracted every candidate phrase pair it could, ended up with bloated tables, and fixed the problem by making candidates compete for the same evidence. We propose to carry that fix over. The judge first aligns each checklist question to the response sentences that support it. Questions and sentences then compete for each other, so redundant questions lose and are pruned. Sentences that no question claims, and that are not relevant to the prompt, are treated the way translation treats unaligned words: as evidence of padding, which lowers the score.

Core Claim

The claim is that a single criterion-by-sentence alignment, built with the cheap judge and filtered by local competitive linking, does two things at once without supervision. Local competitive linking is the rule from Selective Phrase Pair Extraction for Improved Statistical Machine Translation: keep a link only if it is the best-scoring option for its source or for its target. That paper used the rule to halve a phrase table while raising BLEU from 26.78 to 28.30. In the judge setting, the alignment first identifies criteria that are redundant in use, meaning they are repeatedly satisfied by the same evidence across many responses to the same prompt. This is a different notion from criteria that merely look alike as text. Second, the alignment gives an instance-specific precision term: the share of the response that serves neither any criterion nor the prompt.

We expect the second contribution to matter more. We say plainly below that the first may turn out to be matched by simple text deduplication.

Why This Gap Is Real

The judge literature supplies both the motivation and the constraints.

Two papers show the success of checklists but treat the checklist as given. CheckEval expands human seed questions with an LLM and then drops duplicates using an ad-hoc filter. It reports a 0.45 gain in inter-evaluator agreement (Krippendorff's alpha, a standard agreement statistic) and reaches 0.67 for large models. RocketEval has GPT-4o write the checklist once and answers each item independently with Gemma-2-2B or Llama-3-8B using yes/no token probabilities. It reports 0.965 Spearman correlation with human preferences on WildBench at roughly 1/50 of GPT-4o's cost, and it reweights items with a predictor trained on labels. WildBench generates its checklists with unconstrained prompting. None of these three asks whether items overlap in what evidence they consume. RocketEval's reweighting needs labels for every new task. Our pruning needs none.

Two papers show what goes wrong and constrain the design.

RubricRobustness finds that on HealthBench nearly 40% of a response can be replaced with irrelevant sentences before the score drops 25% (α25 = 0.38). Only 4–6% of criteria penalize irrelevance. Adding explicit negative criteria raised padding sensitivity by up to 2.5x. That last result is a strong competitor to our precision term, and we treat it as one.

Judging LLM-as-a-Judge shows two further problems. Verdicts are predictable from rubric text alone with balanced accuracy above 0.8. Judges flip only 37.7% of verdicts when a response is inverted from satisfying to violating a criterion. This suggests that judges answering a criterion against a whole response lean on the criterion's wording rather than the evidence. Forcing the judge to point to a specific sentence should help, and we test whether it does.

The translation side supplies the method and a warning. The method is local competitive linking. The warning comes from A Phrase-based Unigram Model for Statistical Machine Translation, where count-based pruning worked because phrase pairs recur across a corpus. Checklist items are mostly prompt-specific, so pruning must be done per prompt, using many responses to that prompt, not by corpus frequency.

Translation also names the failure we target on the precision side. Improving Statistical Machine Translation with Monolingual Collocation describes the "garbage collection" effect, in which unaligned words get forced onto something. Our design gives padding an explicit null option instead of letting it hide.

Research Program

For a prompt $x$ with criteria $C$ and a response split into sentences $S$, the cheap judge produces a link score $q(c,s)$. This is the normalized probability of "yes" to the question "does sentence $s$, in the context of prompt $x$, provide evidence that criterion $c$ is met?"

Scoring every pair is too expensive, so we cap cost. An off-the-shelf sentence embedder keeps the top three sentences per criterion. The judge then scores those pairs with short inputs. The target budget is close to RocketEval's per-criterion full-response calls, measured in tokens processed.

We add one null node per response: the prompt itself. For each sentence, the judge scores $q(\varnothing, s)$, the probability that $s$ is relevant to the user's request at all.

Linking rule. A link survives if it is the top-scoring criterion for its sentence or the top-scoring sentence for its criterion, following the competitive-linking rule above.

Pruning. Redundancy is estimated across the pool of responses WildBench already provides from many models for the same prompt. A criterion is pruned if, in at least a fraction $\tau$ of responses, every link it would form is beaten by the same other criterion. This is the analogue of a phrase pair that never wins its span. The threshold $\tau$ is set once on a development split.

Scoring. For each surviving criterion, recall $R$ is its RocketEval-style soft "yes" answered against its linked sentences plus the prompt. Precision $P$ is the length-weighted fraction of sentences that are either linked to a surviving criterion or have $q(\varnothing, s)$ above one half. The final score is

$$\text{score} = R \cdot P^{\gamma},$$

where $\gamma \ge 0$ controls how strongly unclaimed, irrelevant text is penalized. Setting $\gamma = 0$ recovers a pruned-but-otherwise-standard checklist. We fix $\gamma$ on development data and report a sweep.

The null node is the design choice that keeps us from punishing thorough answers that contain useful material the checklist did not anticipate. Such material should link to the prompt even when it links to no criterion.

Analysis. Besides the scorer, we measure the redundancy problem itself. We take WildBench checklists and inject controlled duplicates: paraphrases of existing items, and items that are logically entailed by existing ones. We then measure how human correlation and judge variance change. Analysis and System Combination of Phrase- and N-Gram-Based Statistical Machine Translation Systems and Reducing SMT Rule Table with Monolingual Key Phrase found that redundant candidates actively mislead a decoder rather than being harmlessly down-weighted. Whether the same holds for judges is unknown. If redundancy turns out not to hurt, pruning becomes only a cost saving, and the paper's weight shifts to the precision term.

Baselines

The naive reference point is RocketEval-style unweighted checklist scoring with the generated checklist as-is. Each item is answered against the full response by the same Gemma-2-2B and Llama-3-8B judges, and the score is the mean soft "yes." This isolates exactly what our alignment adds. It runs on the same checklists, the same models and the same responses.

The strongest existing method is full RocketEval with its supervised reweighting. RocketEval reports Spearman 0.965 with human preferences on WildBench using Gemma-2-2B. It is the strongest competitor because it already addresses criterion importance, and it does so with labels. We rerun it on our splits with its reported label budget, so that we are comparing against our reproduction and not only the published number. Our unsupervised method counts as beating it if it matches its correlation within confidence intervals while also raising α25 on padded responses. A tie on correlation with large robustness gains would be the expected and sufficient outcome. Beating 0.965 outright is not required.

A holistic GPT-4o grader on WildBench gives an expensive upper reference for correlation and cost. RocketEval reports roughly $3,400 per 1,000 tests for GPT-4o against $71 for Llama-3-8B.

For the precision side, the strongest competitor is the negative-criteria remedy from RubricRobustness. We implement it as an appended instance-specific item, written by the same checklist generator, asking whether the response contains content irrelevant to the request. We also include a plain length-normalization baseline. We beat these if we match their padding sensitivity while penalizing relevant elaboration less, as measured in the experiments below.

Ablations isolate the key ingredients. Pruning by alignment is compared against pruning by text similarity at matched checklist size. Competitive linking is compared against simple thresholding of $q(c,s)$. The precision term is tested with and without the null prompt node. Finally, $\gamma = 0$ is compared with the tuned value. All variants share the judge, the decoding settings, and a tuning budget limited to the development split.

Why not just…?

Why not just deduplicate the checklist by embedding similarity or by asking GPT-4o to remove duplicates? This is a real threat to the pruning half of the proposal. It costs almost nothing and may catch most paraphrase duplicates. Our argument is that redundancy in use differs from textual similarity. "Mentions the deadline" and "gives a concrete timeline" look different but are satisfied by the same sentence. Two items about different JSON fields look alike but test different things. The matched-size ablation settles this. If text deduplication matches alignment-based pruning on correlation and on the duplicate-injection test, we will say so, and the contribution reduces to the precision term and the span grounding.

Why not just add one negative criterion about irrelevant content? RubricRobustness shows that negative criteria raise padding sensitivity by up to 2.5x, so this is the most dangerous alternative on robustness. Its weakness is that a single whole-response question gives the judge no localized evidence. The inertia findings in Judging LLM-as-a-Judge suggest such questions are answered from wording more than content. A single question also cannot easily separate padding from useful elaboration. The paired padded and elaborated perturbation test in the experiments below decides this directly.

Why not just use RocketEval's learned weights, which should down-weight redundant items anyway? Learned weights fit a fixed set of item positions and require labels per task. They cannot see that two items share evidence in a particular response. They also do nothing about content outside the checklist, which is the padding gap. Our method needs no labels. The comparison above tells us whether that freedom costs correlation.

Why not just sample the judge more often to reduce noise? LLM-as-a-Judge on a Budget already shows that adaptive allocation beats uniform resampling. On HelpSteer2 correctness with GPT-4.1 nano, worst-case error at 50k queries was 0.451 for uniform sampling and 0.300 for adaptive allocation. Sampling of either kind addresses variance, not the bias that lets padding go unpenalized. Repeating a biased judgment many times leaves α25 unchanged. The method is orthogonal to ours and could be stacked on top.

Proposed Experiments

Correlation and cost on WildBench. The main benchmark is WildBench, using its prompts, its multi-model response pools for estimating redundancy, and the human-preference correlation protocol RocketEval uses. We report Spearman correlation, the fraction of checklist items pruned, and judge tokens per evaluation.

Agreement on SummEval. Using SummEval with CheckEval-style expanded checklists, we test whether pruning raises inter-evaluator agreement relative to CheckEval's reported 0.67 for large models. This is a setting where heavy expansion should create the most redundancy.

Padding and elaboration robustness. On HealthBench and WildBench we reproduce the RubricRobustness perturbations and report α25 and the negation slope. RubricRobustness reports α25 = 0.38 on HealthBench and negation slopes of −0.38 on HealthBench and −0.55 on WildBench. We pair each padded response with an elaborated control of matched length, in which a second strong model adds relevant, correct detail. A good scorer penalizes the first and not the second. The decisive quantity is the gap between those two penalties, compared across our method, the negative-criterion baseline and length normalization.

Grounding and inertia. We rerun the response-inversion test from Judging LLM-as-a-Judge, where only 37.7% of verdicts flipped. The question is whether answering criteria against linked sentences raises the flip rate. We also check whether pruned checklists are less predictable by a rubric-only classifier.

Decision rule. The proposal is supported if three things hold together. Correlation on WildBench is within confidence intervals of reproduced supervised RocketEval. α25 rises well above the unpruned checklist. The padded-versus-elaborated gap exceeds that of the single negative criterion. If only the robustness results hold, the work stands as a precision method for checklist judges. If text deduplication matches our pruning, we drop the redundancy claim.