Exchange Rates Inside a Single Judgment: Local Surrogates for LLM Judges

When a large language model is used as a judge, scoring or ranking other models' answers, we know from benchmark-wide audits that it is swayed by things that should not matter: length, fake citations, polite tone, which answer comes first. What we cannot yet say is how much those things mattered for this particular verdict. Nor can we say how they trade against substance. For example, we cannot tell how many paragraphs of filler would cancel the penalty for one arithmetic error in a given math solution. This note proposes measuring that trade-off one judgment at a time.

The method takes a single judged answer and produces controlled variants of it. Each variant changes one dial by a measured amount: substance (verified injected errors) or style (neutral padding, fabricated citations, tone). The judge scores every variant, and a small transparent model is fitted to those scores. This small model is a local surrogate: a simple function that mimics a complex model only in the neighborhood of one input, as in the local interpretable model-agnostic explanation method (LIME). Its slopes give an instance-level "exchange rate" between bias and quality. The fitted surrogate can then be checked against the judge's own written rationale, which shows whether the stated reason for a verdict is the real one.

Core Claim

Our claim is that a judge's sensitivity to non-substantive features varies a lot from one item to the next. If so, a single benchmark-wide bias number, such as a flip rate or a global length regression, misjudges which specific verdicts are fragile. We claim that an instance-anchored surrogate fitted over externally defined, graded edit axes can do three things. It can predict the judge's response to unseen combinations of edits better than global or one-axis-at-a-time estimates. It can capture the sharp "one fatal flaw" thresholds that flat linear surrogates average away. And it can expose cases where the judge's chain-of-thought (its written step-by-step justification) cites a reason the surrogate says carried no weight.

Each of these claims can be shown false. If per-item exchange rates turn out to be nearly constant across items, the global audits already suffice, and we will report that.

Relation to Prior Work

Three papers motivate the gap. Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge builds CALM, a suite that injects discrete bias perturbations (longer text, fake citations, swapped order, changed tone) and reports how often verdicts survive across six frontier judges. It finds verbosity and authority bias nearly everywhere. Its edits are on/off and its outputs are aggregate rates. We reuse its style of perturbation but apply it in graded doses and fit a per-instance model.

Articulate Intuition or Genuine Analysis? builds Kahneman4Review, a benchmark of LLM judges grading 3,563 peer reviews. It shows that judges reward analytical form over epistemic function, that they collapse a nine-dimension rubric into one factor, and that length explains about 74% of the score gap between AI-written and human reviews. That finding comes from a global decomposition. We ask the instance-level question it leaves open.

Thinking in a Crowd shows that reasoning traces often rationalize misleading context rather than filter it out. Qwen3-8B fell from 96.22% to 77.03% accuracy with thinking enabled when given misleading snippets. This is why we do not trust the judge's own account of its reasons.

Two papers supply the method. LAMP fits local linear surrogates to LLMs by perturbing the model's self-reported factor weights. It tests classifiers, not judges, and it reports that fit quality degrades in high-confidence "tail" regions where the response surface flattens.

Our work differs from LAMP in two ways. First, our coordinate axes are defined and verified outside the model, so a factor the judge never mentions (such as length) is still measured. Second, the object being explained is an evaluation of another model's output, not a classification. Sparse and Faithful Local Explanations with Piecewise Linear Surrogates introduces PL-LIME, which places a hinge (two linear pieces joined at the explained point). It proves that ordinary LIME slopes are averages of local derivatives that cancel across bends, and it decouples feature selection from fitting to reduce ranking jitter. It was tested only on tabular data. Our edit axes are low-dimensional, typically six to ten, so PL-LIME's scaling limit does not bind and we can reuse its estimator directly.

One paper constrains the design. A Modularized Framework for Explaining Black Box Classifiers for Text Data shows that random token deletion produces ungrammatical text that drives models off their normal input distribution. For that reason we perturb by fluent, verified rewrites rather than by masking.

Research Program

Edit axes. For each judged item (a response, or a response pair in the pairwise setting), we define graded axes on two sides.

The substance side has two axes. The first is the number of injected errors, from zero to three. For GSM8K and MATH we verify each error programmatically by checking that the final answer or an intermediate value changes. The second is the number of removed justification steps.

The style side has three graded axes and one indicator. The graded axes are added neutral padding in tokens, the count of fabricated citations, and a tone level running from curt through neutral to effusive. The indicator, used only in the pairwise setting, is presentation order.

A rewriting model produces the edits. We require an entailment check showing that style edits neither add nor remove claims. Edits that fail verification are discarded. This keeps every variant on the distribution of real answers and tells us exactly what changed.

Sampling and scoring. For each item we draw about 64 joint configurations. Joint sampling is necessary so that interactions and combined effects can be estimated. We record the judge's score as an expected Likert value from token probabilities when these are available, and otherwise as the mean over five samples.

Surrogate. Let $z \in \mathbb{R}^d$ be the edit doses relative to the original item, with $z=0$ the original. Let $s(z)$ be the judge's score. We fit

$$\hat s(z) = s_0 + \sum_{j} \left(\beta_j^{+}\max(z_j,0) + \beta_j^{-}\min(z_j,0)\right) + \sum_{(j,k)\in S} \gamma_{jk} z_j z_k,$$

where the $\beta_j^{\pm}$ are the two one-sided slopes along axis $j$, anchored at the original as in PL-LIME. The set $S$ is a small set of interactions between substance and style axes, and the $\gamma_{jk}$ are their coefficients. We fit with PL-LIME's decoupled shrinkage.

From the fit we compute the local exchange rate

$$\rho_{b} = \beta_{b}^{+} / |\beta_{\text{err}}^{+}|,$$

the dose of bias axis $b$ that cancels one injected error at this item. Interaction terms tell us whether padding hides errors (a negative $\gamma$) or merely adds a constant bonus.

Rationale audit. We parse the judge's chain-of-thought for the original item into the same axes; for example, "penalized for a calculation slip" maps to the error axis. We then compare the axes the rationale cites with the axes whose surrogate slopes are large. A verdict is labeled unfaithful when its rationale cites an axis with no measured slope, or omits a style axis whose slope moves the score by at least half a Likert point within the sampled range.

Baselines

The naive reference point is the global exchange rate. We fit the same surrogate form once on variants pooled across all items, which plays the role of the dataset-level length decomposition in Kahneman4Review, and use it for every item. If local surrogates cannot beat this, the per-item machinery is not justified.

The strongest competitor is a per-item version of CALM's attack-and-detect approach. We apply each axis separately at each dose, measure the score change, and predict combined edits by adding the single-axis effects. This is the cheapest thing a careful auditor would do. It uses the same 64-query budget per item, spent on one-axis-at-a-time doses instead of joint samples. CALM's paper reports Robustness and Consistency Rates, but the file lists no per-model numerical value for them, so no reported result was found on Lacuna for CALM robustness numbers. We therefore re-run this baseline ourselves.

The second competitor is LAMP. Here the judge states its own factor weights, we perturb those weights, and we fit a linear surrogate. LAMP reports R² "comparable to or better than" LIME and SHAP but gives no numerical values on Lacuna. We re-implement it under the same query budget.

The third competitor is standard token-deletion LIME on the judged response, with matched queries.

Two ablations isolate our ingredients. The first replaces the hinged surrogate with a single-slope linear one, which tests whether the anchored hinge matters for threshold behavior such as a judge zeroing out anything with one error. The second drops the interaction terms.

For explanation quality, the only number on Lacuna is PL-LIME's 51% improvement in ranking stability over LIME on the tabular Adult dataset. We treat that as the scale of stability gain worth aiming for. Beating the competitors requires two things. On held-out combined edits, our surrogate must achieve lower prediction error and better verdict-flip prediction (area under the ROC curve) than the additive per-item baseline. And on resampled fits, its axis rankings must be more stable than those of linear LIME and LAMP.

Why not just…?

Why not just ask the judge which criteria it weighted, or rely on its chain-of-thought? The evidence says such self-reports drift from the real decision drivers. Reasoning traces rationalize misleading inputs (Thinking in a Crowd), and judges collapse rubrics into a single "analytical-trace" factor (Articulate Intuition or Genuine Analysis?). Our rationale audit measures this directly. If the self-reported weights agree with the surrogate in the large majority of verdicts, the cheap method wins, and we will say so.

Why not just perturb one axis at a time per item and add up the effects? This is the real threat. If judge responses are additive and roughly linear in dose, the additive baseline is as good as our method and cheaper to explain. The held-out combined-edit test settles it. We expect the additive estimate to fail where padding interacts with errors or where a single error saturates the score. If it does not fail, we will drop joint sampling and recommend the simpler protocol.

Why not just run a global regression of score on length and citations? A global regression is enough if exchange rates are homogeneous across items. We test this by measuring the spread of $\rho_b$ across items and by comparing global and local flip prediction. Low spread would mean the global audits are adequate.

Why not just debias the judge, for example by swapping positions or controlling for length? Debiasing corrects known biases on average. It does not tell a user whether a particular verdict was decided by substance, and it misses biases nobody anticipated. That said, the surrogate's flip predictions can be used to decide which verdicts to re-run, and we compare that targeted re-checking against blanket position swapping.

Proposed Experiments

Ground-truth recovery. We first validate on a planted judge: an open judge model whose system prompt adds a known rule, such as "+1 point if three or more citations" or "score 1 if any arithmetic error." We test whether the surrogate recovers the planted slopes and thresholds. The hinged version should recover the step on the error axis, while the single-slope version should flatten it, which is the LAMP tail-flattening failure.

Real judges. We then run real judges: an open model (Qwen2-72B, as in CALM) with token probabilities, plus GPT-4o and Claude through their APIs. We use about 500 items each from the CALM sources (GSM8K, MATH, TruthfulQA), from the Chat Hard and Reasoning splits of RewardBench in pairwise form, and from Kahneman4Review reviews. For context on judge accuracy, RewardBench reports ArmoRM-Llama3-8B leading with an overall score of 89.0. That number measures preference accuracy, not explanation quality, and we cite it only to place the judges.

Decisive checks. There are four.

The first is held-out prediction. Twenty percent of each item's joint configurations are withheld, and every method predicts score and verdict flips on them.

The second is heterogeneity. We ask whether the spread of $\rho_{\text{padding}}$ across items is large compared with its bootstrap uncertainty.

The third is a test on Kahneman4Review specifically. We ask whether per-review surrogates attribute the reviews' high scores to the padding axis at rates consistent with the 74% global figure, or instead reveal that length dominates for a subset of reviews and is irrelevant for the rest.

The fourth is the rationale audit: the fraction of verdicts whose stated reason contradicts their surrogate. We spot-check it with human annotation of 200 cases.

Cost. About 64 judge calls per item gives roughly 100,000 calls per judge, a cost we report explicitly.