When a language model is used as an automatic grader, it reads a scoring rubric written by a person, such as "give 5 only if the answer is under 50 words", and outputs a score. The field assumes the rubric controls the grade. Work on pre-training task misalignment (the gap between what a model learned to predict during pre-training and what a downstream task asks of it) suggests otherwise. When a task's required labels contradict what pre-training taught the model to associate with a text, even GPT-4 falls to chance. Graders have such built-in associations too, for instance "long, polite, cited, confident answers are good", and these are the same traits behind the verbosity, sentiment, and authority biases catalogued in judge audits. This note proposes to measure how much of an LLM judge's score comes from its built-in notion of a good answer rather than from the rubric. It then tests a training-free correction that reads the judge's own probabilities over score tokens and cancels out the part the rubric does not explain.
The core claim is concrete. Judges will follow rubrics that agree with their priors well and rubrics that contradict them poorly, and this gap is a measurable property of each judge. We expect the gap to predict that judge's susceptibility to style-based biases. A contrastive score, computed by comparing the judge's score distribution under the real rubric with its distribution under a generic "is this a good answer" rubric, should shrink both the gap and the biases while costing little agreement with humans on ordinary rubrics. If the gap does not exist or does not predict bias, the "prior trap" explanation of judge bias is wrong, and that negative result is itself worth publishing.
Prior Work and the Gap
Two results motivate the proposal. Instruction-following Evaluation through Verbalizer Manipulation tests models as classifiers under label words that are natural ("positive"), neutral ("foo"), or inverted ("negative" for positive examples). Zero-shot GPT-4 on inverted mappings performed near 50%, some model families showed inverse scaling, and 4-shot demonstrations lifted GPT-4 from about 58% to about 88%. That paper studies models as task executors on binary datasets. It never examines graders, Likert rubrics, or scoring protocols; we carry its natural/neutral/inverted design into rubric-based judging.
Separately, bias audits describe the symptoms without a mechanism. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge catalogs 12 biases, including verbosity, authority, and sentiment. LLM-as-a-Judge & Reward Model: What They Can and Cannot Do finds that judges score on tone and formatting and miss word-level factual corruption. Pairwise or Pointwise? shows that distractor features flip about 35% of pairwise preferences but only about 9% of pointwise scores. None of these papers asks whether the biases reflect the judge's pre-trained notion of quality overriding the rubric.
The correction borrows its method from C-PMI. C-PMI takes an off-the-shelf model's log-likelihoods and subtracts marginal terms, which removes generic fluency from dialogue scores; on FED this raised average Spearman correlation from 11.9 to 23.9. C-PMI conditions on short hand-written hypothesis strings and uses no rubric. We move the contrast to a different variable, the rubric itself, and apply it to the judge's distribution over score tokens.
Three papers constrain the design. Prometheus produces its integer score as ordinary generated text after a critique, which leaves the score-token distribution available to read. Distributional LLM-as-a-Judge shows that this distribution carries real information: matching it to rater distributions cut SNLI KL divergence from 0.60 to 0.23. Calibrating LLM-Based Evaluator shows that refining rubric wording from 8–12 labeled examples helps a great deal. That makes rubric rewriting a competitor we must beat rather than ignore.
Core Mechanism
For an instruction $x$, a response $y$, and a rubric $R$, let $p(k \mid x, y, R)$ be the judge's normalized probability of emitting score token $k \in \{1,\dots,5\}$ at the score position. $R_0$ is a fixed generic rubric ("rate the overall quality of this response"), and $\lambda \ge 0$ is a weight. The contrastive score is
$$ s_\lambda(x,y,R) = \sum_{k=1}^{5} k \, \tilde p_\lambda(k), \qquad \tilde p_\lambda(k) \propto \frac{p(k \mid x,y,R)}{p(k \mid x,y,R_0)^{\lambda}} . $$
At $\lambda=0$ this is the judge's expected score under the real rubric. At $\lambda=1$ it is a pointwise-mutual-information reweighting: it keeps the evidence for score $k$ that the rubric adds beyond the judge's generic view of quality. The weight $\lambda$ is set on a small labeled golden set, matching AutoCalibrate's budget of 8–12 examples per rubric family. The method needs two forward passes per item and no training. Because pairwise protocols are known to be the more fragile ones, all scoring is pointwise.
Research Program
The first product is a counter-prior rubric suite. We start from Feedback Bench items and IFEval-TweakSet responses and write rubrics whose criteria can be checked by a program: response length, presence of citations, assertive hedging markers, polite formulae, and markdown formatting. Each criterion appears in three versions. The natural version agrees with typical quality priors ("reward complete, well-cited answers"). The neutral version is orthogonal ("score 5 if the response mentions a color"). The inverted version contradicts the priors ("score 5 only if the response contains no citations and is under 40 words"). Ground truth comes from the program, so no human labels are needed for this part. The prior gap of a judge is its accuracy on natural rubrics minus its accuracy on inverted rubrics, measured as whether it ranks the compliant response above the non-compliant one.
The second product is a cross-judge study. We run about ten open-weight judges on the suite, covering Prometheus-13B, Llama-3-8B-Instruct and larger siblings, and Qwen2.5 at several sizes, and we add GPT-4o wherever its API exposes score logprobs. Each judge also runs on CALM perturbations and IFEval-TweakSet distractors. We then test whether the prior gap predicts CALM robustness rates and distractor flip rates across judges, and whether the gap grows with scale within a model family, as the verbalizer paper found for classifiers.
The third product applies the contrastive score to the same judges and benchmarks. We measure what it gains on the counter-prior suite and on bias robustness, and what it costs on human-agreement benchmarks.
Baselines
The naive reference point is the judge's raw pointwise score, taken as the generated integer. We also report the expected score at $\lambda=0$, which separates any benefit of reading the full distribution from the benefit of the contrast itself.
The strongest human-agreement competitor is Prometheus-13B as released. The file reports a Pearson correlation of 0.897 on Feedback Bench's 45 unseen rubrics, against 0.882 for GPT-4 (Prometheus). We apply our correction on top of Prometheus with the same prompts and reference answers. Staying within 0.01 of 0.897 on Feedback Bench while closing the prior gap would count as a success. A larger drop would mean the contrast removes legitimate signal.
For discriminative accuracy we use ArmoRM-Llama3-8B, a Llama-3-8B model with a 19-dimensional rating head and a learned gating network (Interpretable Preferences). The file reports an overall RewardBench score of 89.0 and 76.8 on Chat Hard. ArmoRM cannot take arbitrary rubrics, so we compare against it only on RewardBench and on the verifiable-criterion suite. On the suite we give it the rubric in the prompt, which puts it at a disadvantage we will state openly. We do not expect a training-free correction on an 8B generative judge to beat 89.0 overall. The target is Chat Hard, where style priors should matter most, and beating 76.8 there with a corrected Llama-3-8B judge would count.
The prompt-side competitor is AutoCalibrate-style rubric refinement (Calibrating LLM-Based Evaluator). It gets the same 8–12 golden examples per rubric family that we use to set $\lambda$, and the same judge. The file reports its gains only as relative improvements: 10.4% Spearman over GPT-4 on NewsRoom and 15% over G-Eval-4 on hallucination detection. It reports no absolute number on our suite, so we rerun it ourselves.
The ablations that isolate the key ingredient replace $R_0$ with an empty rubric, with a shuffled rubric taken from another item, and with the real rubric paired with a different response. Only the generic-quality $R_0$ should correct inverted rubrics. If the empty or shuffled versions work equally well, the effect is generic calibration and not removal of a quality prior.
Why not just…?
Why not just give the judge a few demonstrations? This is a real threat. For classifiers, four examples lifted GPT-4 on inverted verbalizers from about 58% to about 88% (Verbalizer Manipulation). We therefore run 4-shot judges on the counter-prior suite with the same golden examples. If demonstrations close the gap as fully as the contrast does, the correction's value shrinks to three things we will state plainly: lower context cost, since rubrics and references already fill long windows in Prometheus-style prompts; and whatever effect remains on CALM biases, which few-shot prompting was never shown to fix. The diagnostic contribution stands in either case.
Why not just fine-tune the judge on counter-prior rubrics? This is also a real threat, because Prometheus shows that rubric-conditioned fine-tuning generalizes to unseen rubrics. We fine-tune Prometheus's backbone on Feedback Collection augmented with automatically generated inverted rubrics, using matched compute, and test it on held-out criterion types. If augmentation wins on those held-out types, the paper's practical recommendation becomes "augment training data", and the contrastive score becomes a tool for judges that cannot be retrained, such as API models. The more interesting question is whether augmentation on length and citation rubrics transfers to criteria it has never seen. The contrast needs no such transfer.
Why not just use chain-of-thought with a frontier model? Existing evidence already argues against this. The verbalizer paper found that chain-of-thought narrowed but did not close the inverted-label deficit. The capabilities study found that GPT-4o and Claude-3.5-Sonnet still miss subtle factual corruption, and AutoCalibrate found chain-of-thought redundant once rubrics were good. We nonetheless include GPT-4o with chain-of-thought on the counter-prior suite so the claim is tested directly rather than assumed.
Proposed Experiments and Decisive Checks
We evaluate on the counter-prior suite, Feedback Bench (Pearson correlation), RewardBench (overall and Chat Hard accuracy), CALM perturbation sets (robustness rate), and IFEval-TweakSet (pointwise score change under distractors). FED is included only for a sanity comparison with C-PMI, whose reported average Spearman is 23.9. The file gives no numerical CALM tables, so for CALM we have no reported result found on Lacuna and all CALM numbers will be our own reruns.
Three results decide the question. The first is whether inverted-rubric accuracy lags natural-rubric accuracy by a large margin in most judges, with the gap widening at larger scales. If there is no gap, the proposal's premise fails. The second is whether a judge's prior gap predicts its CALM verbosity, authority, and sentiment robustness better than its model size does, which would support prior entrenchment as the mechanism. The third is whether the contrastive score at the tuned $\lambda$ closes most of the gap and lowers distractor-induced score shifts while holding Feedback Bench Pearson within 0.01 of the uncorrected judge, and whether it matches or beats 4-shot prompting and AutoCalibrate at the same labeled-example budget. If the empty-rubric ablation matches the generic-rubric version, we will report the method as calibration rather than prior removal.
References
- Instruction-following Evaluation through Verbalizer Manipulation
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
- Pairwise or Pointwise? Evaluating Feedback Protocols for Bias in LLM-Based Evaluation
- C-PMI: Conditional Pointwise Mutual Information for Turn-level Dialogue Evaluation
- Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
- Distributional LLM-as-a-Judge
- Calibrating LLM-Based Evaluator
- Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts