Auditing the Judge: Internal Signals for LLM Reasoning Assurance

Mayank Gulati, Gerhard WunderarXiv 2026 · arXiv

TL;DR

Researchers have found that training "white-box" probes on an LLM's internal attention-head activations can identify errors much more effectively than the model's own self-evaluation scores, especially under tight review budgets. However, these internal signals are highly sensitive to "protocol shifts"—meaning a probe trained for one task or precision level won't necessarily work for another without revalidation.

What Problem Does This Paper Address?

As we deploy Large Language Models (LLMs) in critical reasoning tasks, we often use the models themselves as "judges" to evaluate their own answers. This self-evaluation determines which answers are released to users and which are flagged for costly human review or more expensive re-computation.

The industry standard for evaluating these "judges" is often AUROC (Area Under the Receiver Operating Characteristic curve). However, the authors argue that AUROC is a deceptive metric for real-world deployment. AUROC averages performance across all possible budgets, but in practice, a deployer might only have the budget to review the "worst" 5% of answers.

The core question is: Is the model's verbalized "YES/NO" verdict the best signal we have, or is there better information buried deep within its internal activations?

Methods

The researchers compared two ways of "reading" a model's judgment:

  1. The Black-Box Baseline: They used the continuous probability mass of the "YES" token at the model's verdict position. This is the standard "confidence" score most developers use.
  2. The White-Box Probe: They replayed the judge's prompt and recorded the internal activations of every attention head. Using a technique called Fisher-based head selection, they identified the top 32 heads that best distinguished correct from incorrect answers on a training set. They then trained a simple Multi-Layer Perceptron (MLP) on these activations to predict correctness.

The study utilized a "Budgeted Assurance" framework, specifically measuring error recall (how many mistakes are caught) and residual risk (how many mistakes slip through to the user) at fixed operating points like a 5% review budget.

Fig. 1. The control. (a) The model solves x, producing z = (r, ˆa). (b) (x, z) is rendered into a judge prompt; the black-box baseline reads the YES/NO token mass at the verdict position (Equation (13)). (c) The same prompt is replayed and the pre-projection attention-head state is recorded; the heads with the largest train-only Fisher separability (Equation (16)) are concatenated and, in the controlled ablation, residualised against shallow text statistics (Equation (19)). (d) A probe maps the result to a correctness score, whose ranking drives either the review lever or the compute lever of Section III-B.

Datasets

The study evaluated three open-weight models (Gemma 3 12B, Qwen2.5 7B, and Llama 3.1 8B) across two primary reasoning benchmarks:

  • GSM8K: Grade school math word problems.
  • MATH: Challenging competition-level mathematics.
  • GSM-Symbolic: A variant of GSM8K used for sensitivity analysis to ensure results weren't just due to benchmark memorization.

Key Insights and Ideas

  • AUROC Hides the Truth: The paper proves (Proposition 2) that AUROC is simply an average of error recall over all possible budgets. A probe might have a similar AUROC to a judge but catch significantly more errors at the 5% budget mark—which is the only part of the curve that matters for a resource-constrained deployer.
  • Internal States are Richer than Tokens: The process of "judging" builds a high-dimensional internal state. Compressing this into a single "YES/NO" token discards valuable nuance. The internal probe effectively "audits" the judge's brain to find what it actually "knows" about its mistakes.
  • The "Trust Boundary" is Narrow: Probes are not universal "truth detectors." When the researchers shifted the protocol—changing the dataset, the model's weight precision (e.g., from bf16 to FP8), or the generation budget—the probe's performance dropped significantly. This suggests probes learn the specific "texture" of errors for a given setup.
  • Detection $\neq$ Repair: Identifying that an answer is likely wrong is not the same as predicting that more sampling (like majority voting) will fix it. The probe is better at the former.

Key Results

The internal probe consistently outperformed the judge's token-based score in "in-domain" settings:

  • Error Capture: At a 5% review budget, the probe caught 1.3 to 22.5 additional percentage points of errors compared to the judge (with a median improvement of 7.4 points).
  • Discrimination: As shown in the table below, the probe improved Average Precision (AP) for error detection in all six model-dataset combinations.
Table showing the probe's superior performance in AUROC and Average Precision across different models and datasets.
  • Adaptive Inference: In "routing" experiments, where the score decides which problems get extra compute (5-sample majority voting), the probe significantly beat random allocation. It was particularly effective when the underlying judge was "weak" (e.g., Llama 3.1 8B), as seen in the accuracy gain curves.
Fig. 3. Routing additional inference on GSM8K (k = 5). Each policy sends the indicated fraction of problems to a five-sample majority vote and retains the greedy answer elsewhere; routing fraction m costs 1 + 5m solver calls per problem. The vertical axis is accuracy gain over selecting the same number of problems uniformly at random (Equation (11)); zero is the equal-cost random baseline. Shaded regions are paired 95% bootstrap intervals for the probe and judge. Boxes report their exact comparison at the primary 10% routing budget. The dashed oracle observes which answers sampling will repair or harm and is not deployable. Both panels share a vertical scale, exposing Gemma 3’s smaller available headroom.

Limitations and Future Work

  • Supervision Requirements: Unlike the "black-box" judge score, the "white-box" probe requires thousands of labeled examples from the specific target model and task to train.
  • Lack of Generalization: Because the probes are sensitive to protocol shifts, they must be revalidated whenever the deployment environment changes. They are "deployment audit tools," not general-purpose confidence estimators.
  • Interpretability Ambiguity: It remains unclear if the probe is tracking "objective correctness" or simply the "judge's internal belief." Future work could look at "conflict sets" where the model's belief and the actual truth diverge.
  • Future Direction: The authors suggest using these internal signals during training (e.g., in Reinforcement Learning) to help models better understand and "repair" their own reasoning paths.

Summary updated

Abstract

Language-model systems use self-judgment to decide which answers to release, review, or recompute. Because intervention is costly, an assurance score must rank errors well at the available budget, not merely over the full ranking. We compare the judge's continuous verdict-token score with a supervised correctness probe trained on attention-head activations from the same prompt. Exact identities link fixed-budget error recall to residual risk and show that AUROC averages recall over all budgets. Across three open 7-12B models on GSM8K and MATH, the probe finds 1.3-22.5 additional percentage points of errors at 5% review in all six model-dataset settings (median 7.4 points). At 10% routing, probe-guided allocation improves accuracy over equal-cost random allocation in both GSM8K experiments, but beats judge-guided routing only for the weaker judge. Cross-dataset transfer lowers probe AUROC for every model in both directions relative to target-domain training. Because dataset, weight precision, and generation budget change together, this measures a joint protocol shift rather than a dataset-only effect. Internal signals can improve supervised, in-domain budget allocation, but are not domain-general confidence estimators.
BibTeX
@misc{gulati2026auditing,
  author = {Mayank Gulati and Gerhard Wunder},
  title = {{Auditing the Judge: Internal Signals for LLM Reasoning Assurance}},
  year = {2026},
  eprint = {2611.14323},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.14323},
}
Figure