When a person watches a language model write, they rarely label every sentence. They read along, say nothing while the text looks right, and interrupt when it goes wrong, often a sentence or two late. Current hallucination training uses only a small part of this behavior. Methods either remove the human and rely on automated judges, or ask annotators to grade finished outputs after the fact. Robotics has faced the same problem. Interactive imitation learning puts a human supervisor inside the robot's execution loop. MILE showed that you can model when and why the supervisor intervenes, and then learn from the moments they chose not to. Doing so reaches expert success with far fewer interventions than methods that learn only from the intervened states. We propose carrying that idea to text generation. A model streams its reasoning or summary one step at a time. A supervisor interrupts at errors and types a short correction. The observed pattern of "silent for $k$ steps, then interrupt" is treated as censored evidence about where the first hallucination occurred.
The setting matters because the domains where hallucination is most costly are the ones where automated checking is weakest. On FaithBench's hard summarization cases, the best automated detectors sit between 50% and 60% balanced accuracy, and RAGAS with GPT-4o reaches 62.31% (FaithBench). Synthetic adversaries also fail to produce subtle logical errors such as violated theorem preconditions (Adversarial Training for Process Reward Models). If human attention is the scarce resource, we should extract every bit of signal from it, including the signal in silence.
Core Claim
A single learned "intervention model" can do three jobs that are currently handled by separate, weaker components. It is a head on the language model that predicts the probability that a supervisor would interrupt at each generation step. Trained on streams of interrupts and silences, it serves as a step-level hallucination detector. It also replaces the fixed probability and distance thresholds that CoBa uses to decide when to backtrack. Finally, it acts as a weighting function for updating the model itself, in the style of DAgger: corrections become recovery demonstrations taken from the model's own failing prefixes, and survived steps become down-weighted positive examples.
We claim that, at a matched budget of human time, this outperforms three alternatives: learning only from interventions (the HG-DAgger analogue), grading finished outputs offline (the HADAS analogue), and fully automated on-policy feedback (the RLFH analogue). We expect the advantage to be largest where automated judges are near chance.
What is new relative to MILE is the likelihood itself. Robotic intervention models score the immediate state, and the file notes that this ignores temporal history. Text supervisors, by contrast, react with delay, and hallucinations cascade: an early "cornering" token forces later confabulation, as CoBa and SENTINEL both document. So the intervention event must be modeled as a first-passage time rather than a per-state classification. Let $h_\theta(s_j)$ be the predicted probability that step $j$ (prefix $s_j$) is the first erroneous step. Let $\rho(d)$ be the supervisor's reaction-delay distribution over $d = 0,1,2,\dots$ steps. The probability of observing the first interrupt at step $t$ is
$$P(T=t) = \sum_{d \ge 0} \rho(d)\, h_\theta(s_{t-d}) \prod_{j < t-d} \big(1 - h_\theta(s_j)\big).$$
A stream that ends with no interrupt contributes the survival term $\prod_j (1-h_\theta(s_j))$, discounted by a learned miss rate $\epsilon$ for errors the supervisor overlooks. Here $h_\theta$ is a probit hazard on the policy's hidden states, and $\rho$ and $\epsilon$ are fit per supervisor. This is the text analogue of MILE's mental-model probit. Its extra terms, delay and misses, exist precisely because readers are late and fallible, a limitation the file flags for MILE's assumption of an accurate human.
Research Program
We would build the system in three pieces.
The first is a streaming interface. The model emits one reasoning step or one summary sentence at a time. The supervisor either lets it continue or presses "stop." After a stop, they select the offending span and optionally type a short replacement. As in TIPS, which lets users nudge the state rather than dictate actions, the supervisor may instead choose a coarse semantic nudge such as "claim too strong" or "wrong entity." Constrained backtracking then converts the nudge into tokens.
The second is the intervention model, trained with the likelihood above jointly with the policy. It gets a negative log-likelihood term on corrections and a hazard-weighted imitation term on survived steps. Survived steps far from any interrupt receive weight close to $1-\epsilon$. Steps just before an interrupt receive little weight, because delay makes them suspect.
The third is deployment. The trained hazard head triggers backtracking at test time. Every backtracked branch is logged, along with any human correction, and fed back into the training buffer. CoBa, by contrast, discards these branches after each output.
To measure human cost precisely, the main testbed simulates supervisors from gold step labels. We use PRM800K's human step-level correctness labels on MATH solutions, which cover the "plausible but flawed logic" class of hallucination. The simulated supervisor is given realistic reaction delay, miss rates, and a cost model in which reading, stopping, and writing each have a price. A second testbed uses real annotators on CNN/DailyMail and XSUM summarization. There, entity swaps and fabricated details are the target, and simulation cannot be trusted. The real-annotator study also measures workload on the NASA Task Load Index (TLX), following TIPS.
Baselines
The naive reference is the base policy with greedy decoding and no supervision. We use Flan-T5 for summarization, to match CoBa and HADAS, and a 7B–8B instruction model for reasoning, to match RLFH. Every method below must beat this reference on factuality without collapsing into evasive output. We check that collapse separately by reporting informativeness, since the file identifies the trade between truthfulness and informativeness as a known failure.
The strongest inference-only competitor is CoBa. It backtracks when token probability drops below $\delta$ or when the distance to the source exceeds $\phi$, and it reports FactCC on Newsroom rising from 0.604 to 0.709 with Flan-T5 (CoBa). We run it on the same models with $\delta$ and $\phi$ tuned on the same development split our hazard head uses. Beating it means higher FactCC and AlignScore at equal or lower decode cost, specifically on the subset of confident hallucinations that static thresholds cannot see by construction.
The strongest human-in-the-loop competitor is HADAS. It selects diverse failed summaries for offline human correction and reports FactKB of 87.46 against 69.83 for random selection on CNN/DailyMail with Flan-T5 Small, using 30% of the budget (HADAS). We give HADAS exactly the same annotator-seconds, measured, not estimated, and the same base model. Beating it means higher FactKB and a lower human-judged hallucination rate at equal time.
The strongest automated competitor is RLFH. It decomposes outputs into atomic claims, checks them against a reference, and runs token-level PPO, lifting Llama-3.1-8B FactScore from 0.639 to 0.686 (RLFH). We run it from the released code (AlignRM/RLFH) with the same number of policy updates. For reasoning, the automated competitor is APRM, which reports +3.4 points over ReST-MCTS on math and +5.3 on SciBench out of distribution. The file gives no absolute accuracies, so we rerun APRM ourselves rather than quote a number.
Two ablations isolate our key ingredient. The first, the HG-DAgger analogue, keeps the correction data but drops every non-intervention step. The second keeps non-intervention steps but uses a per-step classifier with no delay or miss terms. If the full survival model does not beat both at matched human cost, the core claim fails.
Why not just…?
Why not just have annotators label every step, as in PRM800K, and train a process reward model? This is the most serious threat. Dense step labels are strictly more informative per step than a censored stream. Our bet is that the streaming protocol costs less human time: silence costs nothing beyond reading, and a supervisor can stop reading after the first error. We also bet that it yields recovery corrections, which dense labels do not. That advantage may be small. The matched-cost experiment below settles it, and if dense labeling wins, the honest conclusion is that silence is not worth modeling.
Why not just replace the human with an LLM judge and run RLFH or APRM? Where judges are reliable, that is cheaper, and we expect it to tie us. But FaithBench's detectors sit near 50–60% balanced accuracy, HaluEval finds ChatGPT at 58.53% on summarization recognition (HaluEval), and chain-of-thought judging degrades verification. We therefore stratify results by how well the judge performs on each item. Our prediction is that gains concentrate where judge accuracy is near chance. If they do not, the human is unnecessary.
Why not just use CoBa, or lookahead decoding, at inference and skip training? CoBa's thresholds cannot flag hallucinations the model is confident about, and it never updates the model, so the same cornering recurs on every query. Lookahead is over ten times slower than CoBa. We test whether the hazard-triggered decoder beats tuned CoBa. We also test whether the fine-tuned policy, run with plain greedy decoding, retains gains, which would show that the correction was learned rather than applied externally.
Proposed Experiments
The decisive experiment uses the simulated supervisor on PRM800K and MATH. We sweep the human-time budget and plot final-answer accuracy and first-error localization F1 for five conditions: our method, the two ablations, a PRM trained on dense labels at equal cost, and APRM. The claim holds if our curve dominates the ablations at every budget and matches or beats dense labeling at low budgets. We then vary the simulated delay and miss rate to confirm that the explicit $\rho$ and $\epsilon$ terms are what buy robustness when supervisors are late or careless.
The second experiment uses real annotators (on the order of a few dozen) on CNN/DailyMail and XSUM, compared against HADAS and CoBa. Metrics are FactCC, FactKB, AlignScore, and a blind expert audit using FaithBench's four labels (Consistent, Benign, Questionable, Unwanted), plus NASA TLX workload. We also test transfer by evaluating the summarization-trained hazard head, zero-shot, as a detector on FaithBench against the 62.31% RAGAS figure. Clearing that figure would show that human-derived silence data teaches detection that automated judges lack.
References
- MILE: Model-Based Intervention Learning
- FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs
- Adversarial Training for Process Reward Models
- Correction with Backtracking Reduces Hallucination in Summarization
- Hallucination Diversity-Aware Active Learning for Text Summarization
- On-Policy Fine-grained Knowledge Feedback for Hallucination Mitigation
- Mitigating Object Hallucinations via Sentence-Level Early Intervention
- Interactive Imitation Learning in State-Space
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models