Seeing but Not Saying: Separating Perception from Readout in LLM Judges

When an LLM judge picks the wrong answer, there are two very different explanations. The judge may never have registered the error: its internal representation of the two responses does not tell them apart. Or it may have registered the error internally and then failed to report it, because something between that internal state and the written verdict, such as a position cue, a fake citation, or an "EXPERT" label, overrode it. Evaluation research built on embeddings has worked through this question for years. A metric computed in an embedding space can only detect what that space encodes, and several papers show that precision and recall go silent when the feature extractor is blind to an artifact (Simon et al.). LLM-as-a-judge work has not asked the question. Nearly all of it measures only the final verdict.

We propose to treat an open-weight judge's hidden states as a feature extractor and audit them the way the embedding-evaluation literature audits Inception or DINOv2. For each kind of judging error, we train a small classifier on the judge's internal activations (a "probe") using ground-truth correctness labels. We then compare what the judge's representation knows with what the judge says. The difference, the readout gap, splits judge failures into blindness and suppression. The two call for different fixes. Blindness requires a different or stronger model. Suppression can be repaired by reading the verdict from the representation. The split also yields a concrete tool: a probe-based verdict that can be checked against injected shortcut cues.

Core Claim

On hard, style-matched pairs, where frontier judges perform near chance, a substantial share of open-weight judge errors are suppression errors, not blindness. The judge's mid-layer activations separate correct from subtly wrong responses better than its verdict does. The shortcut biases documented in the literature act mainly at readout: they flip the verdict without moving the internal quality signal. Both halves of this claim are falsifiable. The first fails if the readout gap is near zero. The second fails if the injected cues shift the probe's output as much as they shift the verdict.

To make this precise, let $a_{\text{probe}}(d,\ell)$ be the held-out accuracy of a probe on layer $\ell$ activations for error domain $d$ (trained on ground-truth labels, never on judge labels). Let $a_{\text{verdict}}(d)$ be the accuracy of the same model's best-prompted verdict. The readout gap is

$$G(d) = \max_{\ell} a_{\text{probe}}(d,\ell) - a_{\text{verdict}}(d),$$

and we report it only after subtracting the accuracy that the same probe reaches on an external sentence embedder. That subtraction is the step borrowed from embedding-dependent evaluation, explained below.

What Prior Work Establishes and Leaves Open

Three results motivate the proposal. JudgeBench built 350 correct/incorrect pairs generated by the same model, so the two responses share a style, and found GPT-4o at 50.86% accuracy. BiasScope injected discovered cognitive biases to build JudgeBench-Pro, where GPT-4o's error rate reaches 74.7%. The Silent Judge showed that provenance and recency cues shift verdicts by up to +30% while the chain-of-thought acknowledges the cue 0% of the time. All three measure only outputs. None can say whether the model "saw" the error. Our proposal adds the missing internal measurement.

The method comes from INSPECTOR. It trains linear or small MLP probes on hidden states of 1–2B models and beats direct prompting of those models by over 20% F1. That result is already evidence that representations can outperform verbal output. INSPECTOR, however, trains its probes on labels from a DeepSeek-V3 teacher, so its probes can only rediscover the teacher's judgments, blind spots included. It also never asks whether a probe on the judge itself disagrees with that judge's verdict. We use ground-truth labels, verified answers and executed code, and we study the judge model directly.

Automated Concept Discovery supplies the geometric lens. It fits sparse autoencoders to differences between external embeddings of paired responses to explain judge preferences. It does not test whether an external embedder's geometry reflects the judge's own perception. Our control condition tests exactly that.

One constraint comes from the embedding literature. Parmar et al. and Salvy et al. show that metrics computed in a feature space pick up incidental structure unless they are carefully controlled and calibrated. For us this means a probe that separates correct from incorrect answers may be reading surface regularities, such as answer length or hedging, and not detecting the error. We therefore require every probe result to be reported against the same probe trained on a general-purpose external embedder that knows nothing about the task.

Research Program

We study three open-weight judges spanning a range of scale: an 8B Llama, a 70B Llama, and Qwen2.5-72B (the model BiasScope used as its teacher). For each pair we build the standard judging prompt in both answer orders. We record residual-stream activations at the final prompt token before the verdict and at the last token of each response, across layers. Probing both orders and averaging removes position bias from the probe by construction.

Probes are logistic regression and one-hidden-layer MLPs. The 350 JudgeBench pairs are too few to train on, so we build a disjoint training pool with the JudgeBench recipe. We sample multiple responses from one model to held-out questions from the same sources (MMLU-Pro, LiveBench, AMC-style math, LiveCodeBench), label them by answer checking or unit tests, and form same-style correct/incorrect pairs. JudgeBench and JudgeBench-Pro are used only for testing.

The analysis has three parts. First, we map $G(d)$ by domain and layer, and ask whether reasoning-heavy domains, math and code, show suppression or blindness. Second, we run a cue-injection experiment. On Silent Judge-style counterfactual pairs, we compare verdict shift with probe shift: the change in probe output when a provenance or recency cue is added. Readout-level bias predicts large verdict shift with near-zero probe shift. Third, we test repair. A probe-read verdict is scored on JudgeBench-Pro to see whether it survives the injected traps that break the verbal judge.

Baselines

The naive reference is the judge's own verdict under a plain pairwise prompt, scored on JudgeBench with order-swapping. The literature reference point for this setting is GPT-4o at 50.86%. Each of our open models is also run under the Arena-Hard judging prompt, which JudgeBench reports lifts GPT-4o to 56.57%. The best of these prompted verdicts for each model is $a_{\text{verdict}}$. Using the best one keeps the comparison from rewarding probes over a weak prompt. Llama-3.1-405B's reported 56.86% gives a scale reference within the same model family.

The strongest existing method is INSPECTOR-style probing with teacher labels. We run it on the same activations, layers, probe capacity, and training pool, with labels from a strong prompted judge in place of ground truth. The only difference from our probe is the label source. This isolates the proposal's main ingredient: ground-truth supervision exposes what the judge's representation knows beyond any teacher. If the teacher-label probe matches ours, the readout gap is not telling us anything about the judge.

The key ablation is the external-embedder control. The identical probe is trained on Sentence-BERT and GPT-2 Large embeddings of the two responses, the extractors used in DCScore and text precision/recall, plus a shuffled-label control for each layer. Only judge-probe accuracy above both controls counts as judge perception.

For absolute context we also report o3-mini's 80.86% and Claude-3.5-Sonnet's 64.29% from JudgeBench. We do not claim a small probe should beat a reasoning model.

Beating the baselines means three things. The probe verdict must exceed the same model's best prompted verdict on held-out JudgeBench with a bootstrap confidence interval that excludes zero. It must exceed both the teacher-label probe and the external-embedder probe. On JudgeBench-Pro, its error increase relative to JudgeBench must be smaller than the verbal verdict's.

Why not just…?

Why not just break down verdict accuracy by error type, as JudgeBench and BiasScope already do? Per-domain verdict accuracy cannot tell a judge that cannot see a flaw from one that sees it and does not report it. GPT-4o at 50.86% is consistent with either. These two explanations call for different interventions, so the internal measurement is not redundant. If $G(d)$ turns out near zero everywhere, the behavioral breakdown was sufficient, and we will report that as the finding.

Why not just use INSPECTOR's probe as is? Its labels come from a teacher LLM, so at best it reproduces a strong judge more cheaply. It cannot find signal the teacher lacks, and its high F1 could hide shared blind spots, which is the circularity problem El Zini et al. raise for evaluations built on representations. The label-source ablation directly measures how much this matters.

Why not just use a reasoning model? This is a real threat. o3-mini reaches 80.86% on JudgeBench and 100% on coding. If long reasoning closes most errors, a probe-based verdict has little practical value on these domains. Our contribution would then be diagnostic: it would show whether reasoning shrinks the readout gap, turning suppression into reporting, or grows representational sensitivity. To distinguish these, we will also probe an open reasoning-capable judge and compare its gap with the non-reasoning variant. Reasoning does not remove shortcut bias, since the Silent Judge's zero cue acknowledgment appeared under chain-of-thought, so the cue experiment remains relevant either way.

Why not just fine-tune a small judge on preference pairs? JudgeBench reports that dedicated fine-tuned judges such as PandaLM and JudgeLM do worse than random guessing on its pairs, which suggests they learn surface preferences. Our probes are a fraction of that cost and are trained on verified correctness, not preferences. They are compared at matched training data against a LoRA-tuned verdict head on the same pool.

Decisive Checks

Four outcomes settle the proposal. First, if the judge probes do not beat the external-embedder and shuffled controls, the "perception" we measured is surface structure and the core claim fails. Second, if the readout gap is positive in math and reasoning, sparing domains where the judge is truly blind, the suppression hypothesis holds for those domains. Third, the Silent Judge replication on ELI5 and LitBench (100 pairs each; reported GPT-4o recency shift +30% on ELI5) tests the location of bias. A cue-induced verdict shift with no significant probe shift places the bias at readout. A matching probe shift places it in perception, and a probe-based verdict would then be no safer. Fourth, the JudgeBench-Pro comparison shows whether reading the representation, rather than the text, gives a verdict that resists injected traps. That is the practical result that would change how open-weight judges are deployed.