Small classifiers trained on a language model's hidden activations, called probes, can often tell whether the model's statement is true even when the model states it confidently. These probes are almost always trained and tested on balanced, curated sets of true and false statements. Real queries are different. Facts about famous entities are abundant, and facts about obscure ones are scarce, and models hallucinate far more often on obscure entities. This proposal asks a narrow question: when a truth probe fails on rare-entity facts, is the hidden-state signal missing, or is the signal present and the probe's last layer simply tuned to the common cases? Long-tailed image recognition gives a precise template for answering this, and it suggests a cheap fix if the second answer is correct.
The template is the finding of Decoupling Representation and Classifier for Long-Tailed Recognition. Under class imbalance, features learned on the natural data distribution are already good. Most of the damage sits in the final classifier, and retraining only that classifier on rebalanced data recovers most of the lost tail accuracy. We transfer this to hallucination probing. We first train a probe encoder on the natural, popularity-skewed distribution of statements. We then freeze it and retrain only the readout with sampling balanced across strata of entity popularity crossed with truth label. We also measure whether this works, and why it fails when it does.
Core Claim
We claim that hallucination probes on frozen language models inherit the long tail of factual knowledge, and that their failure on rare entities splits into two parts. The first part is a readout problem: the probe's decision rule is fitted to head-entity statements. The second part is a representation problem: the truth signal for tail entities is weaker or points a different way. We expect the readout part to be large and fixable by stage-two rebalancing.
The claim matters only if per-bucket thresholding does not already solve the problem, so we take that objection seriously from the start. A binary probe's errors on the tail could just reflect a different base rate of hallucination, and shifting the decision threshold per popularity bucket would fix that. The decisive quantity is therefore within-bucket ranking quality, meaning the probe's AUC (area under the ROC curve) computed only among tail statements. No threshold or per-bucket bias can change it.
Our hypothesis is that within-tail AUC is low under natural training, and that retraining the readout on frozen probe features raises it. A threshold shift cannot do this. If retraining the readout does not raise it, the signal is absent from the activations. That negative result would be useful in its own right, because it would show that the "representational bottleneck" from the decoupling literature applies to language models themselves.
Why This Gap, and What Prior Work Establishes
Several papers motivate the gap. The Internal State of an LLM Knows When It's Lying trains an MLP (a small multilayer perceptron) on frozen activations of OPT-6.7b and Llama-2-7b, called SAPLMA. It reports 71–83% accuracy on held-out topics and an AUC of 0.76 on 245 real model-generated statements. Its training data is balanced, and it never reports how accuracy varies with entity popularity.
HaluEval shows that hallucination rates rise sharply for niche entities such as obscure films and actors, so a deployed probe sees exactly the regime its training set underrepresents. Bold Claims or Self-Doubt? supplies the most important mechanistic hint. Errors on unfamiliar knowledge, as in ALCUNA, are mostly "unaware" hallucinations with low self-consistency. Errors on familiar knowledge are more often "overconfident."
TruthfulQA adds the other side: frequent misconceptions in web text make larger models confidently wrong on head topics. Together these suggest that true-versus-false may look geometrically different in the head and the tail. A single linear direction fitted mostly on head data may not serve both.
The method comes from the decoupling literature. Kang et al. supply classifier retraining on frozen features (cRT), learnable per-group scaling (LWS), and nearest-class-mean classification (NCM). They also make the empirical claim that rebalancing the feature learner itself hurts.
Cross-Layer Attention Probing supplies a stronger probe architecture. CLAP is a small transformer that attends over all layers' activations at the end-of-sentence token, and it reports up to +6.3% AUC over single-layer probes on a Wikidata out-of-distribution test. It naturally splits into an encoder and a readout, which is what decoupling needs. Like SAPLMA, it is trained without regard to entity frequency.
EACL is a precedent from language tasks: freezing a RoBERTa backbone and retraining only the class anchors helped confusable, skewed emotion classes. Our setting differs in that the imbalance lives in a nuisance variable, entity popularity, rather than in the label.
One result constrains the proposal and must be faced directly. The SHROOM shared task found that systems with access to model internals did not beat black-box systems. For example, OPDAI scored 0.836 accuracy on the model-agnostic track and 0.805 on the model-aware track. One reading is that internal signals are overrated. Ours is that internal probes are brittle under distribution shift, and popularity shift is a measurable, controllable part of that shift. We do not claim to resolve the paradox. If our fix works, though, it identifies one concrete reason internal probes underperform in the wild.
Research Program
The first component is a controlled long-tail probing corpus. Following the template style of the True-False Dataset, we generate atomic statements from structured facts about entities such as birthplaces, founders, and chemical properties. Each statement is labeled true or false, and false statements are made by swapping in a plausible wrong object.
Each entity gets a popularity score from an external proxy such as Wikipedia page views, which is our own measurement and not drawn from the cited work. Entities are binned into head, torso, and tail buckets. Because the corpus is generated, we can set the head-to-tail ratio in the probe's training set anywhere from 1:1 to 100:1. We can also build a large, balanced, held-out tail test set and a separate oracle tail-only training pool.
We complement this with natural data. TriviaQA and Natural Questions answers are generated by the model and labeled correct or incorrect, then stratified by the popularity of the question's entity. ALCUNA serves as an extreme-tail stress test.
The second component is the decoupled probe. Stage one trains a CLAP-style encoder plus linear head on activations from frozen Llama-2-7b, Llama 3.1-8B and Gemma-2-9b, using the natural, popularity-skewed sample. Stage two freezes the encoder and retrains the head. Each stratum, defined by popularity bucket and truth label, is drawn with equal probability.
We test three stage-two variants mirroring Kang et al. The first is full head retraining (cRT). The second is a per-bucket learned scale and bias on the logit (LWS); because it only rescales, it can change cross-bucket calibration but not within-bucket ranking, so it doubles as a control. The third is nearest-stratum-mean classification on encoder features (NCM). For the single-layer SAPLMA probe we apply the same recipe, treating the MLP's last hidden layer as the "encoder."
The third component is diagnosis. For every probe we report pooled AUC, within-bucket AUC, and expected calibration error per bucket. The gap between pooled and within-bucket metrics tells us how much of the tail failure is calibration and how much is ranking.
We also measure the angle between linear truth directions fitted separately on head and on tail activations, layer by layer. This tests whether tail truth lives in a different direction. That would explain why a natural-distribution readout fails, and why an encoder with several features gives the retrained head something to recombine. Finally, we split tail errors by the overconfident/unaware typology using sampled self-consistency, to see which type the rebalanced probe recovers.
Baselines
The naive reference point is SAPLMA trained on the natural popularity-skewed sample with a single global threshold. It represents what a practitioner gets today. Its published numbers (71–83% held-out-topic accuracy, AUC 0.76 on real generations, from The Internal State of an LLM Knows When It's Lying) come from balanced data. We found no reported result on Lacuna for popularity-stratified probe performance, so all stratified baseline numbers will be produced by us. Beating it means a statistically reliable gain in within-tail AUC with no more than a one-point loss in within-head AUC.
The strongest existing probing method is CLAP from Cross-Layer Attention Probing for Fine-Grained Hallucination Detection, reported at up to +6.3% AUC over single-layer probes out of distribution. We run it on identical activations and identical training sets, with the same hyperparameter search budget as our stage two, and with the same number of total gradient steps. This rules out the possibility that our stage two wins simply by training longer. Beating it means higher within-tail AUC on both the synthetic tail set and the popularity-stratified TriviaQA/NQ tail at matched head performance.
The per-bucket threshold baseline applies a bucket-specific bias to CLAP's logit, tuned on validation data. It is the cheapest serious competitor. Beating it on pooled accuracy is not enough; the win has to show up in within-tail AUC, which this baseline cannot move.
The rebalanced end-to-end baseline trains the whole probe, encoder included, with stratum-balanced sampling. It is the direct test of whether Kang's finding that rebalancing the representation hurts carries over. Its key ablation partner is our method with the encoder unfrozen during stage two.
The black-box baseline is sampled self-consistency, the signal BAFH builds on in Bold Claims or Self-Doubt?. It is given an equal or larger compute budget, since it needs multiple generations. The oracle upper bound is a probe readout trained on abundant tail-only data over the frozen encoder.
Why not just…?
Why not just shift the decision threshold per popularity bucket? If tail failures are only a base-rate effect, this is sufficient, and it is a genuine threat. The experiment settles it directly. Per-bucket thresholds leave within-tail AUC unchanged, so if the naturally trained probe's within-tail AUC already matches the oracle, thresholding wins and our method adds nothing. We will report that outcome plainly.
Why not just reweight the whole probe during training? Kang et al. found that rebalancing the feature learner degrades representations on ImageNet-LT and iNaturalist. In their setting, decoupled classifier calibration reached 72.5% on iNaturalist against 68.0% for the end-to-end loss-reweighting baseline LDAM+DRW. Whether this transfers to a probe encoder is not obvious. The probe encoder is tiny and sits on an already-frozen model, so reweighting might be harmless here. The rebalanced end-to-end baseline and the unfrozen-encoder ablation decide this, and if they tie our method, the simpler recipe should be preferred.
Why not just feed entity popularity to the probe as an input? A probe that sees popularity can learn bucket-specific offsets. Unless it has enough tail examples, it still fits its truth direction mostly to the head, so this is essentially the threshold baseline learned end to end. We include it as a variant of the threshold baseline. It also requires a popularity estimate at deployment, which stage-two rebalancing needs only during training.
Why not just use retrieval or self-consistency on the tail? Black-box sampling may genuinely be better on tail entities. Bold Claims or Self-Doubt? finds that tail errors are mostly low-self-consistency hallucinations, which sampling detects well. We therefore compare against it under matched compute and report the cost per query. A single-pass probe that comes close in tail AUC at a fraction of the cost is useful. A probe that falls well short indicates that the tail signal is not in the activations.
Decisive Checks
Four checks decide the outcome. The first is that within-tail AUC of the natural-distribution CLAP probe is clearly below the tail-only oracle; if not, there is no readout problem to fix. The second is that frozen-encoder stage-two retraining closes a substantial share of that gap while per-bucket thresholding closes none of it. The third is that the head-versus-tail truth-direction angle grows in the layers where the natural readout fails, which gives a mechanistic account. The fourth is that the gains persist as the training imbalance ratio grows from 1:1 to 100:1 and transfer from the synthetic corpus to popularity-stratified TriviaQA, NQ and ALCUNA.
If the oracle itself is near chance on ALCUNA-style tail facts, the program yields its alternative result: the truth signal for knowledge the model barely has is not linearly present, and no probe recalibration can supply it.
References
- Decoupling Representation and Classifier for Long-Tailed Recognition
- The Internal State of an LLM Knows When It's Lying
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- Bold Claims or Self-Doubt? Factuality Hallucination Type Detection via Belief State
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Cross-Layer Attention Probing for Fine-Grained Hallucination Detection
- Emotion-Anchored Contrastive Learning Framework for Emotion Recognition in Conversation
- SemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
- OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data