Which Judge Biases Can Robust Attention Remove?

An LLM judge is a language model asked to grade or compare other models' answers. These judges get swayed by text that should not matter. A tag saying an answer came from an "expert" or was written in 2025 flips verdicts, and the judge's written explanation never mentions the tag. Most fixes act outside the model. They strip the cue, swap answer order, or fit a regression afterward. Those fixes only work when you already know what the cue is and can cut it out.

A separate line of work rebuilds attention itself. Attention is the step where each token averages information from the other tokens. These methods make that average a robust one, meaning a few extreme tokens cannot dominate it. Such operators can be dropped into a pretrained model without retraining. We propose to put them inside open-weight judges. The goal is not to show that this "fixes bias." The goal is to sort judge biases into two kinds: those it can remove and those it cannot.

The organizing hypothesis comes from robust statistics. A robust average caps the influence of tokens whose content sits far from what the rest of the context agrees on. It does nothing about tokens the model has learned to weight heavily. So the method should help with planted claims inside a response that clash with the response's own content. Examples are a self-declared "verified by experts," a fake citation, a recency boast, or a misleading aside. It should not help with position bias, or with authority tags the model has simply learned to trust. Showing which biases fall on which side, with a measurement that predicts this in advance, is the contribution.

Core Claim

Harnessing Robust Statistics for Trustworthy AI shows that the aggregation step of attention is the solution of a weighted least-squares problem: it is the point that minimizes squared distance to the value vectors, weighted by attention scores. Squared penalties let a single far-away token pull the solution arbitrarily far. ProTransformer supplies the working method. It replaces the squared penalty with Huber or minimax-concave penalties. The Huber penalty grows only linearly for large errors. The minimax-concave penalty flattens out entirely, so the most extreme tokens get zero weight. The resulting problem is solved with a few iterations of reweighted least squares. The method works plug-and-play on LLaMA and Vicuna, where it improved prompt-attack accuracy on LLaMA by 17.8% and cut the AdvBench jailbreak success rate on Vicuna to 1.8%. ProTransformer never tests judges, long contexts, or bias.

Our claim is twofold. First, plug-and-play robust attention will lower the verdict shift caused by embedded distractor cues inside candidate responses, without lowering agreement with humans on clean pairs. Second, how much it helps for a given bias type will be predicted by a pre-registered diagnostic. That diagnostic is the share of attention-weighted residual mass carried by the cue tokens in the unmodified judge, where the residual is $\|V_j - V^*\|$, the distance of cue token $j$'s value vector from the attention output $V^*$. Cues that score high are "outlier-type." Cues that score low are "learned-prior-type," and robust attention should leave them unchanged.

Position bias, documented in Judging LLM-as-a-judge with MT-Bench and Chatbot Arena, serves as a planned negative control. The hypothesis predicts no improvement there, because position bias does not live in a few extreme tokens.

Motivation and Constraints From Prior Work

Two papers motivate the gap.

The Silent Judge injects provenance and recency labels into ELI5 and LitBench pairs. It finds verdict shifts up to +30% for GPT-4o, with a Cue Acknowledgment Rate of 0%. That is, no judge explanation ever cites the cue it followed. This rules out asking the judge to reason its way out.

Thinking in a Crowd shows the same failure in reasoning models. With two misleading snippets, Qwen3-8B falls from 96.22% to 77.03% with thinking on, and responses get longer as the model rationalizes the noise. This constrains us: a fix must act before the reasoning trace, not through it.

Noise-Response Calibration treats the judge as a black box and measures how its performance falls as inputs are corrupted. It finds flat slopes on tabular tasks, where judges rely on memorized shortcuts. This is a constraint too. It tells us some judge failures are learned priors rather than context outliers, which is exactly the case we predict robust attention cannot touch.

The external fixes are what we must beat or complement. Length-Controlled AlpacaEval regresses out length after scoring. Trust or Escalate abstains and escalates uncertain verdicts to stronger judges. Noisy but Valid and Doubly-Robust LLM-as-a-Judge correct judge noise statistically. None of them changes what the judge reads. All of them need the confounder to be named in advance, or need human calibration labels.

Research Program

Benchmark. We first build an embedded-cue benchmark. The source pairs come from three places: pairwise examples from ELI5 and LitBench following the Silent Judge protocol, and MT-Bench pairs that have human expert votes. Into one response of each pair we write a cue inside the answer text, so it cannot be removed by deleting a metadata field. The cue families are:

  • self-asserted authority ("as a board-certified physician…");
  • fabricated citations;
  • recency claims;
  • self-grading ("this answer is complete and correct");
  • length padding;
  • a misleading factual aside adapted from SciAux snippets (SciAux).

We also keep the original Silent Judge metadata-tag condition and a position-swap condition.

Each cue is placed in the better response on half the pairs and in the worse response on the other half. This separates "the cue helps" from "the cue hurts." The main outcome is verdict shift rate, the fraction of verdicts that change when the cue is added. Accuracy is measured against the human label. The Cue Acknowledgment Rate is reported for continuity with prior work.

Judges and intervention. The judges are open-weight instruction models in the 7–8B range from the LLaMA and Qwen families, since closed judges cannot be modified. We will also try one larger open model if compute allows.

We apply ProAttention with Huber and minimax-concave penalties. There are three configurations: all layers, middle layers only, and only the attention over the candidate-response spans. A small development split tunes the robustness threshold. Every baseline gets the same tuning budget.

Diagnostic. Before running any intervention, we compute the outlier-share diagnostic for each cue family in the unmodified judge. We record it in advance, so the prediction is made before the result is seen.

Baselines

Unmodified judge. The naive reference is the same open judge with standard softmax attention and the Silent Judge prompt. It is run on identical pairs with greedy decoding.

A second, sanity reference is ProAttention with a squared penalty. This should reproduce softmax attention and confirms the solver adds no effect of its own.

Strongest external competitor. This is the cue-agnostic debiasing stack that a practitioner would actually deploy:

That regression raised Spearman correlation with Chatbot Arena from 0.94 to 0.98 and cut verbosity gameability from 25% to 10%. It is the strongest reported number in the file for judge debiasing.

For verdict shift under embedded cues on open judges, no result was found on Lacuna. The closest figure is the +30% recency shift for GPT-4o on ELI5 from The Silent Judge.

Cheaper attention-level ablations. These isolate the key ingredient. The robust penalty may matter, or any reduction of extreme influence may be enough. We test:

  • clipping value-vector norms;
  • top-k sparse attention;
  • lowering the softmax temperature in the same layers.

Each is tuned on the same development split.

What counts as winning. ProAttention beats the field if it meets three conditions:

  • it reduces verdict shift on outlier-type cue families by a margin larger than the external stack and the cheap ablations, measured on held-out pairs;
  • it keeps clean-pair human agreement within two points of the unmodified judge;
  • its gains stack with the external stack rather than being redundant with it.

Why not just…?

Why not just strip the cue from the prompt? For metadata tags this is the right answer, and we concede it. The Silent Judge condition is kept only as a reference.

The benchmark targets cues written into the response itself. There, stripping requires a detector that already knows every shortcut. Because the Cue Acknowledgment Rate is 0%, the judge cannot tell us which shortcut it used. If a simple claim detector paired with deletion matches ProAttention on embedded cues, we will report that robust attention is unnecessary.

Why not just debias the scores afterward? Post-hoc regression in Length-Controlled AlpacaEval needs a measured confounder such as token count. It leaves the written critique unchanged.

For padding it is a genuine threat. We expect it to match or beat ProAttention there, since padding is not an outlier in value space. For fabricated citations or self-grading, no simple covariate exists to regress on. The experiment shows directly which families each method covers.

Why not just tell the judge to ignore such claims, or let it think longer? Thinking in a Crowd shows that extended reasoning rationalizes misleading context rather than discarding it. The Silent Judge shows explanations hide the cue. An ignore-instruction is nonetheless part of the strongest baseline. If it closes the gap on its own, the proposal fails.

Why not just sharpen or sparsify attention? This is the most serious alternative. Temperature and top-k attention are far cheaper than ProTransformer's 1–2× latency overhead. They are run as matched ablations. If they equal the robust penalty, the honest conclusion is that "cap extreme influence" is the mechanism and the M-estimation form is incidental.

Proposed Experiments and Decisive Checks

Experiment 1: the predictive test. Across cue families and judges, we correlate the pre-registered outlier-share diagnostic with the reduction in verdict shift from ProAttention. A clear positive rank correlation supports the theory. Position bias and metadata-tag authority should show low outlier share and near-zero gain. If position bias improves substantially, the mechanism story is wrong and something else is happening.

Experiment 2: no harm. Clean MT-Bench pairs with human votes test whether the robust judge discards legitimate content. The concern is real: ProTransformer notes that a badly chosen threshold can suppress genuine semantic variation. We report agreement as a function of the threshold. We also run a SciAux reader task to check that the effect transfers to the reasoning setting, measuring whether misleading-snippet accuracy moves back toward the clean 96.22% for Qwen3-8B.

Experiment 3: complementarity. We combine ProAttention with the external stack and with the selective escalation of Trust or Escalate. We then measure coverage at a fixed human-agreement target. If robust attention raises the fraction of pairs a small judge can keep without escalating, it has value even where it does not remove bias outright.

The result we would consider decisive in either direction is a clean split. The prediction is that robust attention removes most of the embedded-claim shift and none of the learned-prior shift, with the diagnostic sorting the two in advance.