One way to build a model that marks which sentences in an LLM's output are false is a self-training loop. The current annotator labels a large pile of unlabeled responses, a new annotator is trained on those labels, and the cycle repeats. This is the Expectation-Maximization (EM) pattern: alternately guess the hidden quantities (here, the labels) and refit the model given those guesses. The best documented loop of this kind, ANAH-v2, makes its labels trustworthy by brute force. For every sentence it draws 32 candidate annotations and takes a majority vote. The dataset expansion cost about 100 hours on 32 A100 GPUs. Most of that compute goes to the labeling step, and most of it is likely spent on sentences where all 32 samples agree.
We propose to replace most of that sampling with a small prediction head that reads the annotator's hidden states once and predicts what the 32-sample vote would have been. Full sampling is kept only for the minority of sentences where the head is unsure, plus a small random audit set. Because the annotator changes after every training round, the head is refit each round on that round's audited samples, so it tracks a moving target. The claim is that this keeps label quality and final annotator accuracy at roughly the level of the 32-sample loop while cutting labeling compute severalfold. The proposal also says plainly when a cheaper trick, such as simply drawing fewer samples, would do just as well.
Core Claim
The specific new element is an amortized E-step: a learned, single-pass estimate of the self-consistency vote. It is recalibrated every round against a sampled audit set and paired with an adaptive rule that decides where to spend sampling budget. Single-pass truthfulness readouts already exist. Token-level density methods fit a linear regressor on Mahalanobis distances across decoder layers and add only 5.3% to 7.6% overhead over one generation pass. HalluTracer averages per-layer truth logits at the prompt boundary. Both are trained once against fixed labels and used as stand-alone detectors.
Our setting differs in two ways. First, the head's target is the annotator's own vote distribution, which changes as the annotator improves. A head fit once goes stale, and we expect that staleness to be measurable. Second, the head is used inside the loop as a labeling oracle, where its errors feed back into training. The confirmation-bias concern raised by Expectation Maximization Pseudo Labels therefore applies directly. That paper shows pseudo-labeling is EM and that a learned, per-sample acceptance threshold beats a fixed one, which motivates our learned routing threshold.
Concretely, for sentence $x$ the head outputs $\hat p_t(y \mid x)$, a distribution over annotation labels $y$ at round $t$. It is trained to match the empirical vote frequencies $\bar v_t(y \mid x)$ obtained from $K$ samples on the audit set $A_t$:
$$ \mathcal{L}_t = -\sum_{x \in A_t} \sum_{y} \bar v_t(y \mid x)\, \log \hat p_t(y \mid x). $$
A sentence is routed to full $K$-sample voting when the entropy of $\hat p_t(\cdot \mid x)$ exceeds a threshold $\tau_t$. Otherwise it takes the head's argmax label. $\tau_t$ is chosen each round as the smallest value for which the head agrees with the $K$-sample majority at least $1-\epsilon$ of the time on held-out audit items below the threshold. Here $\epsilon$ is a user-set label-disagreement tolerance. This gives a direct, checkable guarantee about how far the cheap labels may drift from the expensive ones.
Research Program
The base system reproduces the ANAH-v2 recipe as faithfully as the paper allows: a 7B InternLM2 annotator, the phased annotation format (does the fact exist, extract the reference, judge the hallucination type), the ANAH seed data, and $K=32$ sampling with majority vote. On top of this we build three components.
The first component is the head itself. It is a linear or two-layer map from PCA-reduced, layer-averaged hidden states of the annotator at the start of the annotation. The design borrows directly from the multi-layer pooling of Vazhentsev et al. and the depth averaging of HalluTracer. We read the annotator's states, not the generator's. This matters for the SHROOM observation that access to the generating model's weights did not help detection. Our signal asks a different question: "will my own sampled judgments agree?" That is plausibly much easier to read from activations than whether the original output was true.
The second component is the per-round audit-and-route procedure above. The audit fraction (we will sweep 2% to 10%) is the main cost knob.
The third component is an analysis of drift. We measure how quickly a head trained at round $t$ loses agreement with the round-$t+1$ annotator's votes. This determines whether per-round refitting is needed at all.
Two failure modes are known in advance. HalluTracer's authors note that prompt-onset readouts miss errors that build up during generation. In our case the annotator generates a multi-step reasoning trace, so we will also test a variant that reads hidden states after the first phase (fact existence), at modest extra cost. CUPID reports that plug-in heads degrade when representations collapse. We will check whether head accuracy drops as the annotator is fine-tuned over rounds.
Baselines
The naive reference point is the round-0 annotator trained on seed data only, with no self-training. Every other method must clear it.
The strongest existing method is the full ANAH-v2 loop with $K=32$ sampling on every sentence. We run it with identical seed data, the same unlabeled pool, the same number of rounds, and the same M-step training schedule. We report both its accuracy and its measured GPU-hours. The paper reports 89.55% accuracy on the ANAH-v2 internal test set for the 7B annotator, against 86.97% for GPT-4. It also reports 81.54% zero-shot on HaluEval and 94.44% zero-shot on HalluQA (ANAH-v2). Our reproduction's numbers become the reference. Beating this baseline means matching its accuracy within one percentage point on all three test sets while using at most one third of its E-step compute. Exceeding its accuracy at equal compute, by reinvesting the savings in more unlabeled data, would be a stronger result.
The compute-matched uniform baseline runs the same loop with $K$ reduced uniformly, to 4 or 8, so that total E-step compute equals ours. This is the most dangerous competitor, discussed below.
The one-shot weak-labeling baseline follows OPDAI. It labels the pool once with consistency filtering and trains once, with no further rounds. On SHROOM, OPDAI reports 0.836 accuracy in the model-agnostic track and 0.805 in the model-aware track. Top systems reported between 0.81 and 0.85 (SHROOM).
Two ablations isolate the key ingredient. The first is a frozen head trained at round 0 and never refit; it tests whether tracking the moving annotator matters. The second keeps the head but routes randomly instead of by head entropy, holding the fraction of sampled items fixed; it tests whether the head's uncertainty is informative.
All variants share tuning budgets. The routing threshold and audit fraction are tuned only on ANAH validation data.
Why not just…?
Why not just draw fewer samples, say four instead of 32? This is a real threat. If most sentences are easy, a 4-sample vote may already match a 32-sample vote, and no head is needed. Our answer has to be empirical. On a fixed set of sentences we compute the 4-, 8- and 32-sample majorities and measure their agreement. In the full loop we compare final annotator accuracy at matched GPU-hours. If uniform small $K$ reaches the same accuracy-compute curve, the head adds nothing and we will report that. We would then pivot to the cheaper finding that ANAH-v2's 32 samples are oversized. Our bet is that disagreement concentrates in a minority of sentences, which a small uniform $K$ under-samples and adaptive routing targets.
Why not just train a single-pass probe once, on sampled votes, and stop? This is the frozen-head ablation. It would suffice if the annotator's hidden geometry and vote behavior barely change across rounds. The drift measurement settles it. Large drop-off in frozen-head agreement after one or two rounds justifies per-round refitting; small drop-off means the simpler static version wins.
Why not just ask the annotator for verbalized confidence? The file's evidence is against it. BSDETECTOR reports 0.831 AUROC on GSM8K for self-reflected certainty, versus 0.951 when combined with sampled consistency. SELFDOUBT finds verbalized proxies inconsistent across models. We still include verbalized confidence as an alternative routing signal, so the comparison is direct rather than assumed.
Why not just use a fixed confidence threshold on the head? EM Pseudo Labels shows fixed thresholds invite confirmation bias in self-training. Our threshold is recalibrated each round against audited votes. We run a fixed-threshold variant to measure whether that recalibration matters in this domain.
Proposed Experiments
The primary experiment runs three or four self-training rounds on the ANAH seed and unlabeled pool for every method above. For each, we report accuracy on the ANAH-v2 test set, HaluEval and HalluQA, along with total E-step GPU-hours. The deliverable is an accuracy-versus-compute curve per method.
The secondary experiment transfers the procedure to SHROOM. We use its 60,000 unlabeled items as the pool and its 1,000 validation items as the seed and audit calibration set, and report test accuracy against OPDAI's 0.836. We will also report agreement with the fraction of annotators who marked each item, since about 30% of SHROOM items split the annotators. A head that predicts vote distributions should be judged on soft targets, not only on hard labels.
Three checks are decisive. First, at matched compute, the adaptive head must beat uniform small-$K$ sampling on final accuracy; if not, the idea is not worth its complexity. Second, the per-round head must beat the frozen head on agreement with the current annotator's votes. Third, entropy routing must beat random routing at the same sampling fraction. Together these show where the savings come from, and whether amortizing the E-step is a real gain or just a cheaper description of using fewer samples.
References
- ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models
- Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models
- HalluTracer: Hallucination Detection via Depth-Averaging Truth Signals
- Expectation Maximization Pseudo Labels
- SemEval-2024 Task 6: SHROOM, a Shared-task on Hallucinations and Related Observable Overgeneration Mistakes
- CUPID: A Plug-in Framework for Joint Aleatoric and Epistemic Uncertainty Estimation with a Single Model
- OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data
- Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness
- SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratio