Large language models are now routinely used as automatic graders ("LLM judges") that read two candidate answers and say which better follows an instruction. These judges are easy to fool. A response that is polite, long, and neatly formatted but ignores the instruction often wins. The most reliable fixes split the judgment into stages: score each answer on its own before comparing, or rate separate aspects and add them up outside the model. These fixes work, but they multiply the number of model calls for every verdict. The standard way to make a model robust, adversarial training (repeatedly searching for inputs that fool the model and training on them), is even more expensive for text, because each search can take thousands of queries.
This note proposes to pay the cost of the staged judge once, offline, and use it to train a cheap judge. A staged judge does more than produce better labels. Through its learned aspect weights, it also says which properties of an answer should not move the verdict: tone, length, formatting, filler. Those properties define a set of allowed perturbations, the text analogue of the small "epsilon-ball" in image adversarial training. We generate a handful of edits to each answer along those irrelevant aspects, pick the one the current judge is most fooled by using only forward passes, and train a small number of extra parameters on a frozen judge so that its single-pass verdict ignores such edits. It should still flip when an edit breaks instruction compliance. The goal is a one-call judge that matches the robustness of multi-call staged judges on the LLMBar adversarial benchmark, with training cost measured in hundreds of steps rather than a full adversarial-training run.
Core Claim
The perturbation set for adversarially training a judge does not need to be found by expensive search. It can be read off a decomposed evaluator. If the aspect-weighting stage of a decomposed judge assigns near-zero weight to an aspect, edits that change only that aspect are nuisance perturbations. Edits that degrade a high-weight aspect, such as instruction adherence, are label-flipping perturbations. Training on the worst of a few such edits per example turns the inner maximization of adversarial training into a cheap ranking over a few candidates. Distilling the decomposed judge's verdicts at the same time recovers most of its accuracy in one forward pass. The new element is using decomposition as a threat model, not only as an inference procedure. Neither the decomposed-evaluation papers nor the adversarial-training papers in this file do that.
Prior Work and Positioning
Three papers motivate the problem and supply the target accuracy. Evaluating Large Language Models at Evaluating Instruction Following built LLMBar, a benchmark of answer pairs where the polished answer is the wrong one. It found that strong judges fall near chance, and that chain-of-thought prompting can hurt because the judge invents reasons to prefer the polished answer. Its fixes are prompt-level only. PRePair scores each answer in isolation before comparing and gains large accuracy on LLMBar. DnA-Eval generates aspects, scores and weights them, and sums the result with an external calculator rather than asking the model to do the arithmetic. Both papers name their multi-call cost as a main limitation. We keep their outputs as an offline teacher and remove them from the inference path.
HD-Eval supplies a key method piece. It fits a small transparent aggregator over sub-criterion scores using as little as 5% of human preference data, and it uses permutation importance to identify inactive criteria. We reuse that aggregator and importance analysis to label aspects as nuisance or decisive. HD-Eval itself stops at better aggregation and never trains the judge.
On the cost side, Robustness Tokens freezes a vision transformer and trains about ten appended vectors with an invariance loss on clean inputs plus a correction loss on attacked inputs. It converges in about 200 steps. We borrow that two-part loss and the frozen-backbone recipe. Its threat model is pixel-norm attacks, while ours is semantic edits. Towards Improving Adversarial Training of NLP Models (A2T) makes text adversarial training cheaper by replacing word-importance search with one gradient pass and adversarially perturbing only a subset of data. That is the closest efficiency idea. A2T still searches word substitutions against a fixed label, while we draw whole-answer rewrites from an aspect-defined set and rank them with forward passes.
Adversarial Training for Process Reward Models trains a grader against a learned error generator in a two-player reinforcement-learning game. It reports gains along with heavy compute and generator-capacity limits. We avoid a learned adversary entirely.
Two papers constrain the design. A Coin Flip for Safety shows that safety judges hover near chance under attack and that ensembles fail because blind spots transfer across judges. It also shows that best-of-N search "hacks" judges, inflating measured attack success. Any judge we train must therefore face an adaptive best-of-N search against itself, not only fixed test sets. RobustJudge shows that input filters such as retokenization cut attack success but cost clean accuracy and latency. It also shows that judge-tuned models are comparatively resilient, which makes plain fine-tuning a serious competitor (addressed below).
Research Program
Let $s_\phi(x, a, b)$ be the probability that a frozen judge, with a small set of trainable parameters $\phi$, prefers answer $a$ over $b$ for instruction $x$. The construction has four steps.
First, run a decomposed teacher offline on a pool of instruction–answer pairs drawn from open models on prompts disjoint from LLMBar. The teacher is a PRePair-style pointwise stage followed by DnA-style aspect scoring and external weighted summation. Fit an HD-Eval-style aggregator on a small human-labeled slice. Its permutation importances split aspects into a decisive set $D$ (instruction adherence, factual correctness) and a nuisance set $N$ (length, tone, formatting, politeness).
Second, for each training answer, use one rewrite call each to produce $K$ nuisance edits, which change one aspect in $N$ while preserving content. Also produce $K$ breaking edits, which degrade one aspect in $D$ while improving polish. Check each rewrite with the teacher's pointwise stage. Discard nuisance edits that the teacher's decisive-aspect scores flag as changed, which is a guard against the meaning drift A2T reports.
Third, train $\phi$ with
$$ \mathcal{L}(\phi) = \ell\big(s_\phi(x,a,b), y\big) + \lambda \max_{a' \in N_K(a)} \ell\big(s_\phi(x,a',b), y\big) + \mu \max_{a'' \in D_K(a)} \ell\big(s_\phi(x,a'',b), \bar y_{a''}\big), $$
where $y$ is the teacher (or human) verdict, $\ell$ is cross-entropy, $N_K(a)$ and $D_K(a)$ are the $K$ nuisance and breaking edits of $a$, and $\bar y_{a''}$ is the teacher's verdict on the edited pair. The verdict on a breaking edit is usually flipped, but not always, so we use the teacher's judgment. Each max is a ranking of $K$ forward passes under the current judge, so one training step costs about $2K{+}1$ forward passes and one backward pass. The edits are regenerated only at the start of each epoch, following A2T's subset schedule.
Fourth, compare two parameterizations at matched trainable-parameter count. One is appended soft tokens in the Robustness Tokens style. The other is low-rank adapters, which add small trainable low-rank matrices to the frozen weights. The claim is about the training signal, so neither parameterization is assumed to win.
We will report training GPU-hours, inference calls, and generated tokens per verdict next to every accuracy number.
Baselines
The naive reference is the backbone judge prompted for a direct pairwise verdict, Llama-3.1-8B-Instruct. PRePair reports 48.59% LLMBar accuracy for this setup. It costs one call and sets the floor.
The strongest existing method at this backbone size is PRePair on the same Llama-3.1-8B-Instruct, reported at 70.84% LLMBar accuracy in the same paper. We will rerun it ourselves with the released prompts on all LLMBar subsets so that the numbers come from one evaluation harness. We will count its calls and tokens. Our method beats it if it matches or exceeds PRePair's accuracy on LLMBar-Adversarial, and especially on the hand-built Manual subset, within one call. A result more than two points below PRePair at one call is a cost–accuracy trade we would report as such, not a win.
The full decomposed teacher (PRePair plus DnA-style aggregation) is the upper reference our student distills from. Here we can only state that DnA-Eval reports gains of up to 39.6% across benchmarks. No reported LLMBar-only number was found on Lacuna for DnA-Eval, so we measure it ourselves.
For context with a far larger model, GPT-4 with Rules, Metrics, and Reference reaches 82.8% on LLMBar-Adversarial according to Evaluating Large Language Models at Evaluating Instruction Following. We do not expect an 8B judge to beat this, and we include it only as an anchor.
Three ablations isolate the key ingredient, and all of them use the same backbone, the same teacher-labeled pool, the same trainable parameters, and the same step budget, with tuning budgets matched by giving every variant the same number of hyperparameter trials.
The first is distillation only, which sets $\lambda=\mu=0$. It asks whether the teacher's labels alone carry the robustness.
The second trains on random rather than worst-of-$K$ edits. It asks whether the cheap inner maximization matters or plain augmentation suffices.
The third replaces the aspect-defined edits with generic paraphrases. It asks whether the decomposition-derived threat model matters.
Why not just…?
Why not just run PRePair or DnA-Eval at inference and accept the cost? If calls and latency are free, that is the right choice, and our method only matters if it closes most of the accuracy gap at one call. Both papers name inference cost as their main limitation, and judges used inside reward modeling or best-of-N selection are called millions of times. The comparison against PRePair at matched backbone, reported as accuracy against tokens per verdict, settles whether the trade is worth it.
Why not just fine-tune the judge on teacher labels? This is the real threat. RobustJudge found judge-tuned models notably resilient, so distillation alone may capture most of the gain. The distillation-only ablation tests this directly. If it matches the full method on fixed test sets, the remaining question is the adaptive best-of-N attack described below. If distillation also holds up there, we would report that the adversarial terms add nothing and that the contribution reduces to a cost-efficient distillation recipe.
Why not just prompt the judge with chain-of-thought or robustness instructions? LLMBar's authors found that chain-of-thought lowers accuracy on adversarial pairs because the judge rationalizes the polished answer. PRePair found that judges spot the flaw in their reasoning and then overrule it. Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design shows that zero-cost "inputs may be perturbed" instructions help image models. We include a matched instruction ("ignore tone, length and formatting; judge only instruction compliance") as a one-call baseline so this is measured rather than assumed.
Why not just ensemble several judges or filter inputs? A Coin Flip for Safety shows that ensembled judges share blind spots and are collectively wrong on hard cases, and an ensemble also multiplies calls. RobustJudge shows that retokenization and detector models cost clean accuracy and add latency. Neither addresses polished-but-wrong answers, which are fluent text that a perplexity filter passes.
Proposed Experiments
The primary test is LLMBar (princeton-nlp/LLMBar), reported separately on the Natural set and on each Adversarial subset (Neighbor, GPTInst, GPTOut, Manual). GPTOut answers are model-written distractors that resemble our generated edits, so we treat Manual as the cleanest held-out test and flag any gain that appears only on GPTOut as possible leakage. We also report positional consistency by swapping answer order, since a single-pass judge could gain accuracy by exploiting order.
The decisive check is an adaptive judge-hacking attack in the style A Coin Flip for Safety describes. For each LLMBar instruction, a best-of-N search rewrites the losing answer for polish alone, using the same kinds of nuisance edits, and keeps the rewrite our judge most prefers. We measure how often the verdict flips to the losing answer as N grows, against the distillation-only judge, PRePair, and the base judge. This is the experiment that separates adversarial training from distillation. The method succeeds only if its flip rate grows clearly more slowly with N than the distillation-only judge's does. Because soft tokens kept private act like a key, as the Robustness Tokens authors note, we run the attack both with black-box access and with full access to $\phi$, scoring rewrites by our judge's own probabilities.
A secondary transfer test applies the heuristic injection attacks from RobustJudge to its text tasks. That paper reports 80–100% attack success for heuristic attacks on standard judges and 16.67% after retokenization. These attacks were never seen in training, so this tests whether aspect-defined training extends beyond its own edit family.
As a stretch probe of how far the threat model reaches, we score the frozen-judge variant on JudgeStressTest from A Coin Flip for Safety. Its reference point there is JailJudge's AUROC of 0.48 under GCG-REINFORCE on Llama-3.1-8B. We expect little transfer, because safety harm is a different decisive aspect than instruction compliance. A null result would show that the nuisance/decisive split has to be re-derived for each domain.
Throughout, the cost ledger counts teacher labeling calls, rewrite calls, training GPU-hours, and per-verdict inference calls. A reader can then judge whether the one-time offline cost is repaid after a stated number of evaluations.
References
- Evaluating Large Language Models at Evaluating Instruction Following
- The Comparative Trap: Pairwise Comparisons Amplifies Biased Preferences of LLM Evaluators
- DnA-Eval: Enhancing Large Language Model Evaluation through Decomposition and Aggregation
- HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Towards Improving Adversarial Training of NLP Models
- Adversarial Training for Process Reward Models
- A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
- LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
- Improving Adversarial Robustness in Vision-Language Models with Architecture and Prompt Design