When a language model is used to grade another model's answer for factual errors (an "LLM judge"), it reads the answer cold. It has no record of similar cases it has graded before, and no way to say "this looks unlike anything I have seen, so I may be wrong." We propose to give an open-weight judge that record. We store the judge's internal activations for a few thousand candidate answers whose correctness humans have verified. For a new answer, we look up the most similar stored cases in that same activation space. The verified labels of those neighbors give a verdict, and the distance to them says how far the judge is extrapolating. Only the judge needs to be open. The model that produced the answer can be a closed API.
This matters for two reasons. First, prompted judges are weak and unreliable on exactly the cases that matter. ChatGPT detects hallucinations in summaries with only 58.53% accuracy on HaluEval (HaluEval). Frontier judges also change verdicts on irrelevant cues while never mentioning them (The Silent Judge). Second, the current fix of putting retrieved neighbors into the judge's prompt costs several extra judge calls per item. It also breaks when the neighbors share the candidate's error. Distance in a network's internal representation has already been shown to track error well outside language (A quantitative uncertainty metric controls error in neural network-driven chemical discovery). Our bet is that the judge's own activations are the right space in which to do this for factuality.
Core Claim
We make two claims and test them separately. The first is about ranking. A nearest-neighbor vote over the judge's mid-layer activations, using human-verified labels, detects hallucinated answers as well as or better than the judge's own verdict. It also matches or beats inserting text-embedding neighbors into the prompt, at one forward pass instead of $k+1$.
The second claim carries the novelty. The distance to those neighbors predicts when the judge itself is wrong, and does so better than the judge's verdict probability or its stated confidence. This enables selective judging: the judge abstains on a controlled fraction of items and the error rate on the rest drops predictably. A trained linear probe can be expected to match the first claim. It has no built-in counterpart to the second, and the experiments are designed around that distinction.
Concretely, let $h(x)$ be the judge's hidden state at layer $\ell$, read at the token just before it emits its verdict on candidate $x$. Let $N_k(x)$ be the $k$ stored reference items whose hidden states are closest in cosine distance $d$, with verified labels $y_j \in \{0,1\}$. We define a verdict score and a reliability score:
$$ s(x) = \frac{\sum_{j \in N_k(x)} w_j\, y_j}{\sum_{j \in N_k(x)} w_j}, \qquad u(x) = \frac{1}{k}\sum_{j \in N_k(x)} d\big(h(x), h(x_j)\big), $$
where $w_j = \exp(-d(h(x), h(x_j))/\tau)$ and $\tau$ is a temperature tuned on validation data. The score $s$ is used for ranking. The reliability score $u$ is used for abstention, and it plays the role that latent space distance plays in the chemistry work.
How this sits relative to prior work
The closest work is SAGE (Detecting Errors in AI-Generated Annotations: When and Why Semantic Neighbors Help). SAGE retrieves neighbors with an off-the-shelf text embedder and pastes them into the judge's prompt as reference anchors, reporting +3.8 to +9.7 AUROC points on HaluEval. It supplies our motivation and our main competitor. Its own analysis states a "usability requirement": the anchors must be correct. When errors cluster, the anchors share the error mode and the mean-shift benefit collapses.
We differ in three ways. Neighbors are found in the judge's internal space rather than a separate embedder. What we use from each neighbor is its verified label, not its text. And we need no extra judge calls. Because the labels come from verification rather than from the candidate generator, correlated generator errors do not contaminate them. That is a testable claim, not an assumption.
The method itself is borrowed from two places. One is LaSeNN (Improving classifier decision boundaries using nearest neighbors), which smooths image-classifier predictions with $k=3$ latent neighbors. The other is latent space distance as an error controller (A quantitative uncertainty metric controls error in neural network-driven chemical discovery), which cut mean error from 8.6 to 3 kcal/mol by keeping the most certain 25% of predictions. It also found 4,200 reliable predictions on QM9 where ensembles found 74. Neither has been applied to a generative judge assessing text.
Two results constrain the design. Park et al. (Steer LLM Latents for Hallucination Detection) find that pretrained LLM latents cluster by topic and fluency rather than truth. If that holds inside a judge, raw neighbors would be topical rather than truth-bearing. We therefore treat their truthfulness separator vector as a candidate transformation of the judge's space, not as a competitor alone. INSIDE (INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection) shows that middle layers carry the truthfulness signal in the generator. That guides our layer search, but INSIDE needs 10–20 sampled generations and white-box access to the generator, both of which we avoid.
The Latent Critic (Actionable Hallucination Detection) shows that residual-stream signals can be read in under 10 ms, but it critiques only its host generator on tool calls. We judge third-party free text. Halu-J (Halu-J: Critique-Based Hallucination Judge) is the strongest text-only trained judge. We use its base model family so that comparisons are clean.
Research Program
We build the judge on Mistral-7B and Llama-3.1-8B, both open. Each sees a fixed hallucination-judging prompt. For every reference and test item, we record its hidden states at every fourth layer at the pre-verdict token. Reference pools come from the labeled training splits of HaluEval (QA, dialogue, summarization), TruthfulQA, and ME-FEVER, with pool sizes swept from 256 to the full split. Retrieval uses exact search at these sizes. FAISS is needed only for the scaling study.
The analysis has three parts. First, we map which layers make neighborhoods truth-bearing rather than topical. We measure label purity of neighborhoods as a function of layer, with and without a separator vector trained on 32 labels in the manner of Steer LLM Latents for Hallucination Detection. Second, we fit risk-coverage curves for $u(x)$ against competing reliability signals. Third, we test robustness in two settings: cross-dataset transfer (pool from one benchmark, test on another), and a correlated-error split. In that split, test candidates come from TruthfulQA misconception categories, where TruthfulQA documents that models converge on the same falsehood. This is precisely where SAGE's text neighbors should be wrong together.
A secondary analysis connects to the shortcut-bias problem. We inject the provenance and recency cues from The Silent Judge into ELI5 pairs. We then ask whether $u(x)$ rises when a cue flips the verdict, which would make cue-driven verdicts detectable from geometry even though the rationale is silent. We consider this exploratory, and the core claims do not depend on it.
Baselines
The naive reference point is the judge's own verdict probability on the same prompt with no retrieval. For reliability, we also use its verbalized confidence. For orientation, HaluEval reports ChatGPT detection accuracies of 62.59% (QA), 72.40% (dialogue), and 58.53% (summarization) (HaluEval). We will rerun our open judges under identical prompts rather than borrow these figures.
The strongest existing method on the exact question is SAGE (Detecting Errors in AI-Generated Annotations). We run it with the same judge model, the same reference pool, and $k=3$, the value that paper reports captures most of its gain. Its reported number is a gain of +3.8 to +9.7 AUROC points over the unanchored judge on HaluEval. No absolute AUROC was found on Lacuna for SAGE on HaluEval or TruthfulQA, so we reproduce it. Beating SAGE means matching its AUROC within one point on HaluEval at roughly one-quarter of its judge calls, and exceeding it on the correlated-error split.
On ME-FEVER, the strongest trained text judge is Halu-J, reported at 91% accuracy versus 83% for GPT-4o and 78% for base Mistral-7B (Halu-J). We place our method on base Mistral-7B and compare to that 78% and to the 91% ceiling. We do not claim to beat Halu-J outright without fine-tuning. We also test whether our abstention signal, applied on top of Halu-J's own activations, raises accuracy at fixed coverage.
The trained-probe baseline is a logistic probe, plus the separator-vector method of Steer LLM Latents for Hallucination Detection. That method reports 84.2% AUROC on TruthfulQA with Llama-3.1-8B, though as a generator-side detector. Both are trained on the same judge activations and the same labeled pool, with a matched hyperparameter search budget.
Two ablations isolate the key ingredient, which is the space and the labels. The first replaces judge activations with text-embedding neighbors while keeping the label vote. This separates "where you search" from "how you use neighbors." The second keeps judge-space neighbors but votes with the judge's own predicted labels instead of verified ones, which tests whether verification is what protects against correlated errors.
Why not just…?
Why not just train a linear probe on the judge's hidden states? This is the real threat for ranking. A probe trained on the same labels may well match $s(x)$ in AUROC, and we will report it if it does. The proposal does not rest on beating the probe at ranking. It rests on $u(x)$ giving a reliability estimate that a probe's output probability does not. The chemistry work found that confident parametric models are "unanimously" wrong off-distribution while distance is not (A quantitative uncertainty metric controls error). The cross-dataset risk-coverage experiment decides this. If probe confidence matches $u(x)$ there too, the neighbor method reduces to an interpretable, retraining-free variant of a probe, and we will say so.
Why not just use SAGE-style prompt anchors with a better embedder? Prompt anchors require the neighbors to be correct and to be similar in the way that matters. TruthfulQA shows that errors on misconception questions are shared across models, so anchors retrieved by surface meaning will often carry the same falsehood. The correlated-error split and the text-embedding ablation measure this directly. Cost is the other issue: SAGE needs $k$ extra judge passes, whereas we need none.
Why not just use the judge's verdict probability or self-consistency as confidence? The Silent Judge shows verdicts shifting by up to 30% on irrelevant cues with 0% acknowledgment (The Silent Judge). That makes stated or output-level confidence a poor guide to when the judge is wrong. Self-consistency adds many sampled passes, and the Latent Critic paper reports semantic entropy at over 12 seconds per check (Actionable Hallucination Detection). Our risk-coverage comparison includes both and will show whether they are actually worse.
Why not just fine-tune the judge, as Halu-J does? Fine-tuning helps, as Halu-J's 91% shows, but it fixes the judge's knowledge at training time and gives no per-item reliability. The reference pool can be extended with new verified cases without retraining, much as GRACE edits a model by adding latent keys (Aging with GRACE). We test the two as complements, not substitutes.
Decisive Experiments
We report four metrics. AUROC measures hallucination detection on HaluEval subsets, TruthfulQA, and ME-FEVER. The area under the risk-coverage curve and the error rate at 75% coverage measure selective judging; the 75% figure mirrors the chemistry paper's "keep the most certain quarter" protocol in reverse. Expected calibration error is computed for $s(x)$.
The first decisive check is the layer-purity map. If no layer of either judge gives neighborhoods with label purity clearly above the base rate, even after the separator transform, the idea fails at its premise and we stop.
The second is transfer. The pool is built from TruthfulQA and the test set is drawn from HaluEval QA, and the reverse. For context, the separator vector transfers from TruthfulQA to TriviaQA at 79.8% AUROC on the generator side (Steer LLM Latents for Hallucination Detection). The claim survives only if $u(x)$ gives a lower risk-coverage area than probe confidence and verdict probability under shift.
The third is the correlated-error split. Judge-space neighbors with verified labels should hold up there while SAGE degrades. If both degrade equally, our second ablation will show whether the verified labels or the search space is responsible.
Finally, we report wall-clock overhead per item against SAGE and a self-consistency judge, so that the cost argument rests on measurement rather than on counting forward passes.
References
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- A quantitative uncertainty metric controls error in neural network-driven chemical discovery
- Detecting Errors in AI-Generated Annotations: When and Why Semantic Neighbors Help
- Improving classifier decision boundaries using nearest neighbors
- Steer LLM Latents for Hallucination Detection
- INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection
- Actionable Hallucination Detection: Translating Latent Uncertainty into Agentic Critique
- Halu-J: Critique-Based Hallucination Judge
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors