Automated judges are language models or learned reward models that score or compare other models' answers. They are now the main source of training signal and leaderboard rankings, and they are swayed by things that should not matter: how long an answer is, which answer is shown first, whether it carries a tag like "EXPERT" or "NEW (2025)". Current fixes target one bias at a time. They swap the order of the two answers, tell the judge to ignore length, or call a second critic model, and each fix only covers the bias it was built for. Speech research hit the same problem decades earlier. There, the answer was to train the model's internal features so that a separate "spy" network cannot tell what background noise was present. That work found a surprising side effect: features made blind to the noises seen in training also held up under noise types never seen in training.
This note proposes to test whether the same transfer holds for judges. We train a reward model whose internal representation of an answer is pushed, adversarially, to carry no information about how the answer is rendered (its length padding, markdown layout, tone) while the content stays fixed. We then measure whether that model also resists shortcuts it was never trained against, such as provenance and recency tags, sentiment manipulation, and fake consensus. The comparison that decides the question is against the simple alternative a reviewer would propose first: putting exactly the same rewritten answers into training as ordinary labeled examples. If representation-level invariance transfers to unseen shortcuts and plain augmentation does not, the field gains a general way to harden judges instead of a growing list of per-bias patches. If it does not transfer, we learn that judge debiasing has to stay bias-specific. Both outcomes are worth knowing.
Core Claim
The claim is that an evaluator trained to be invariant to a family of content-preserving surface variations will show smaller verdict shifts on held-out shortcut cues than an evaluator trained on the same variations as labeled data. The mechanism is that adversarial removal of nuisance information compresses the features the scoring head can read, and that compression also strips other content-irrelevant signals.
The evidence motivating this comes from speech. Adversarial Speaker Verification trained speaker features to be blind to specific training distortions and found that they generalized to unseen acoustic environments. A Joint Noise Disentanglement and Adversarial Training Framework for Robust Speaker Verification, called NDAL, combined three pieces: a gradient-reversal adversary, a loss pulling together clean and noisy embeddings of the same speaker, and a reconstruction path. It cut VoxCeleb1 equal error rate from 5.84% to 3.88% and evaluated on unseen noise sets. We reuse NDAL's structure: paired renderings of the same content play the role of clean and noisy recordings of the same speaker.
How This Differs From Existing Work
Three lines of work sit near this idea, and none of them changes the judge's own representation in a way meant to transfer across biases.
The first line works at inference time or after the fact. Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector routes each verdict to a separately trained critic that explains possible bias and asks the judge to reconsider. It reports an average 18.5% accuracy gain. It leaves the judge's features untouched, adds about 1.5 seconds per example, and runs into "stubborn" frontier judges that refuse to change their verdict. UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge reweights Elo updates across ten judges using a consensus anchor. It raises human correlation from 0.65 to 0.81, but it needs a panel of judges and can amplify biases the panel shares. Both papers motivate our work by showing that biases persist after prompting. Neither paper is a substitute for it, because neither yields a single judge that is itself robust.
The second line supplies the method. MABR: Multilayer Adversarial Bias Removal Without Prior Bias Knowledge places gradient-reversal adversaries at several encoder layers of BERT and RoBERTa classifiers to remove dialect and gender signal. Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation treats clean and speech-recognition-corrupted text as two domains and aligns them with a Wasserstein-GAN discriminator, gaining 2% exact match on Spoken-SQuAD. Neither paper was applied to evaluators. Neither tested transfer to nuisance types absent from training, and that transfer test is our central question.
The third line provides the measurement targets. The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge shows recency tags shifting GPT-4o verdicts by +30% on ELI5 while judges never mention the cue. Verbosity Bias in Preference Labeling by Large Language Models quantifies length bias. JudgeBench: A Benchmark for Evaluating LLM-Based Judges shows that fine-tuned judges such as PandaLM and JudgeLM fall below chance once style is controlled. JudgeBench constrains the design: naive fine-tuning on preference data learns surface heuristics, so any training intervention has to show it does not make this worse.
Research Program
The evaluator is a pairwise reward model: a decoder backbone of roughly 8B parameters with a scalar scoring head, trained with the standard Bradley–Terry preference loss, matching the scale of the strongest classifier reward models in RewardBench: Evaluating Reward Models for Language Modeling. We use a classifier reward model rather than a generative judge because the adversary needs a well-defined feature vector feeding the decision. A generative-judge variant, with the adversary attached to the hidden state at the verdict token, is a secondary extension.
Rendering domains. For every response in the preference training set, we generate $K$ content-preserving renderings with a strong rewriting model. The renderings are: concise, padded with non-informative elaboration, bulleted markdown, flowing prose, and confident versus hedged tone. Each rendering is checked for content preservation by a bidirectional entailment test against the original, and renderings that fail are discarded. This check is the weakest link in the design. We will report its rejection rate and hand-audit 200 renderings.
Training objective. Let $h(x, y)$ be the backbone's final-layer feature for prompt $x$ and response $y$, and $r(x,y)$ the scalar score. Let $y^{(k)}$ denote rendering $k$ of a response, and let $D$ be a discriminator predicting $k$ from $h$. We train
$$ \mathcal{L} = \mathcal{L}_{\text{BT}}(r) + \lambda_c \, \mathbb{E}_{k,k'} \lVert h(x,y^{(k)}) - h(x,y^{(k')}) \rVert^2 - \lambda_a \, \mathcal{L}_{\text{CE}}\big(D(h(x,y^{(k)})), k\big), $$
where $\mathcal{L}_{\text{BT}}$ is the preference loss on the original pairs, the second term pulls together renderings of the same content (NDAL's feature-robust loss), and the third term is applied through a gradient-reversal layer. The discriminator therefore minimizes its own cross-entropy, while the backbone is updated to maximize it. $\lambda_c$ and $\lambda_a$ are weights. Following MABR, we also attach discriminators to two intermediate layers.
Pairing matters for one reason. Because the adversary only sees renderings of identical content, it cannot remove genuine quality differences, such as a longer answer that is longer because it is more complete. This directly addresses the "negative transfer" risk listed for adversarial invariance.
Seen versus held-out nuisances. Training uses only length, layout, and tone renderings. Every evaluation cue is held out from training: provenance and recency tags (Silent Judge ELI5 and LitBench), sentiment manipulation (RBD's ScienceQA setup), bandwagon or fake consensus (RBD's Chatbot Arena setup), and presentation order. Position is held out because a pointwise reward model is order-free by construction. We include a pairwise-concatenated variant of the reward model specifically so that position bias can be measured.
Mechanistic check. After training, we freeze the features and fit linear probes for both seen and held-out cues, for example "does the input contain an EXPERT tag." If transfer is real, decodability of the held-out cues should fall in the invariant model. MABR's layer-wise drop in domain accuracy is the precedent for this kind of probe. This tells us whether any behavioral transfer comes from feature compression or from something incidental.
Baselines
The naive reference point is the same backbone trained with the Bradley–Terry loss alone on the original preference data, with no renderings. It shows how much bias an ordinary reward model has on each probe. The JudgeBench finding that PandaLM and JudgeLM fall below chance suggests this baseline may be poor on style-controlled correctness.
The strongest competitor on the exact question is counterfactual augmentation. The same model is trained on the same original data plus every accepted rendering, entered as extra preference pairs that inherit the original label, with the same number of gradient steps. This is the baseline the proposal must beat, because it sees exactly the same information about which variations should not matter and differs only in enforcing that at the label level instead of the representation level. We give it the same hyperparameter search budget (an equal number of trials over learning rate and augmentation ratio) as the adversarial model gets over $\lambda_c$ and $\lambda_a$. All runs use three seeds.
Two ablations isolate the key ingredient: consistency loss only (no adversary) and adversary only (no consistency loss). If consistency alone matches the full model, the minimax game, with its known instability, is unnecessary and we will say so.
For the strongest existing methods we report published numbers as reference points. They are not matched comparisons, since they use different backbones. On RewardBench, ArmoRM-Llama3-8B reports an overall score of 89.0 (RewardBench). On JudgeBench, Claude-3.5-Sonnet reports 64.29% accuracy and o3-mini (high reasoning) 80.86% (JudgeBench). On held-out recency cues, GPT-4o shows a +30% verdict shift rate on ELI5 (The Silent Judge). RBD reports an 18.5% average accuracy gain across its bias suites but gives no absolute per-suite figure in our file. We will rerun RBD's critic on top of our naive and invariant models instead of comparing to its reported gain.
Beating the strongest competitor means three things together. First, verdict shift rates on the held-out cues fall below the augmentation baseline by more than the seed-to-seed spread. Second, linear-probe decodability of held-out cues falls with them. Third, RewardBench overall and JudgeBench accuracy fall by no more than one point. JudgeBench has only 350 pairs, so differences under roughly five points are within sampling noise. We will report bootstrap intervals and will not claim gains inside them.
Why not just…?
Why not just augment the training data with the rewrites? This is the real threat. It may well match the invariant model on the seen nuisances (length, layout, tone), and we expect it to. Augmentation teaches the scoring head to give the same score to these specific variations, but nothing pushes the features to discard other content-irrelevant cues. The adversarial model is pushed, at least in principle, toward features with less room for any such signal. The held-out-cue comparison is designed to settle exactly this. If augmentation transfers equally well, the adversary adds nothing and we will report that as the main result.
Why not just strip metadata tags and swap answer order at inference? For tags and order this works and is cheap. We do not claim to beat it on those cues in deployment. We use them as held-out probes of whether invariance generalizes, because a deployment can never list every shortcut in advance. RBD's sentiment and bandwagon manipulations are examples of cues that cannot be removed by deleting a token. Tone and length cannot be stripped without changing the answer, and Verbosity Bias in Preference Labeling by Large Language Models reports bias values of 0.328 (GPT-4) and 0.428 (GPT-3.5) even under order swapping.
Why not just prompt the judge or add a critic like RBD? Prompting is documented as too weak. The Silent Judge finds large verdict shifts under explicit instructions, and the Arena-Hard prompt moves GPT-4o on JudgeBench only from 50.86% to 56.57%. The RBD critic is stronger, but it pays for every evaluation in latency and depends on the judge agreeing to be corrected. The two approaches are complementary. We test RBD on top of both the naive and the invariant reward model to see whether a more robust base model leaves the critic less to fix.
Why not just regress out length from the scores? A post-hoc length correction handles one scalar nuisance. It cannot address layout, tone, or unseen tags, and it penalizes answers that are long because they are complete. We include it as a cheap reference on the verbosity probe only.
Proposed Experiments
Training data is a standard human preference set (HH-RLHF, which the verbosity study also uses) with renderings added. Evaluation runs on five sets. RewardBench overall and Chat Hard check that accuracy is preserved. The Chat subset is watched specifically, since this is where legitimate detail could be wrongly suppressed. JudgeBench tests style-controlled correctness. The Silent Judge ELI5 and LitBench sets, at 100 pairs each, measure verdict shift rate for recency and provenance tags. The RBD bias suites (GSM8K verbosity, ScienceQA sentiment, Chatbot Arena bandwagon) measure held-out shifts. MT-Bench human-agreement pairs check that agreement with people stays near the reported ~80% level for GPT-4.
The decisive check is a two-by-two comparison: augmentation versus invariance, crossed with seen versus held-out cues. The interesting outcome is a large gap on held-out cues with no gap on seen cues. A second decisive check is a dose-response test: increasing the adversary weight $\lambda_a$ should monotonically lower held-out cue decodability and verdict shift until RewardBench accuracy starts to fall. That trade-off curve is what a practitioner would use to choose an operating point. Training instability, which is a known risk of gradient-reversal training, is tracked by reporting discriminator accuracy over training and the fraction of seeds that collapse.
References
- Adversarial Speaker Verification
- A Joint Noise Disentanglement and Adversarial Training Framework for Robust Speaker Verification
- Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector
- UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge
- MABR: Multilayer Adversarial Bias Removal Without Prior Bias Knowledge
- Mitigating the Impact of Speech Recognition Errors on Spoken Question Answering by Adversarial Domain Adaptation
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Verbosity Bias in Preference Labeling by Large Language Models
- JudgeBench: A Benchmark for Evaluating LLM-Based Judges
- RewardBench: Evaluating Reward Models for Language Modeling