Language models write left to right, so whether a token is a hallucination (a fluent but false statement) often only becomes clear from what comes after it. A birth year is wrong only once the sentence around it settles which person and which event it belongs to. Recent work shows that models "know" they have erred a few tokens too late. Their internal states flag an error more reliably shortly after emitting it than at the moment of emission. Current fixes work around this lag at inference time. One approach rolls generation back after the fact. Another simulates the future with short look-ahead drafts. Both treat the model's timing as fixed.
We propose to shorten the lag during training. We fine-tune the generator on two objectives at once. The first is ordinary next-token prediction. The second is an auxiliary head that, at step $t$, predicts whether the span being started will later be judged false. That judgment is made with the full future text visible. The future text is a training-only signal, like expert annotations a classifier learns from but never sees at test time. Because the auxiliary loss updates the backbone and not just a probe on frozen states, the model is pushed to represent "this is about to go wrong" before committing. At decoding time, one extra linear readout per step then supports choosing among candidate tokens before commitment, with no rollouts and no rollback.
Core Claim
The delay in hallucination awareness is at least partly learnable, not a fixed property of causal models. If so, training on a future-informed factuality target will move the point where hallucinations become detectable earlier in the sequence. That in turn makes cheap preventive decoding competitive with, or better than, backward rollback and forward lookahead at equal latency. The new ingredient is not a better detector. It is jointly training the generator's own representation so that information currently available only from later tokens becomes available at the current one.
Where This Sits in Prior Work
The motivating evidence comes from three papers on hallucination timing. DSVD reports "delayed awareness": small MLP probes on hidden states detect errors better a few tokens after they occur. DSVD exploits this by rolling back $r$ tokens and resampling with a penalty. Enhancing Hallucination Detection via Future Context shows that a hallucinated sentence is followed by further hallucination 80–90% of the time. It detects errors by sampling future sentences with a separate model. HALLUCANA forks generation at high-entropy steps, rolls each branch forward $N$ tokens, and vetoes branches with a small classifier.
All three accept the causal model's representation as given and pay at inference for missing future context. DSVD pays in retrospective rollback, the future-context detector in multiple drafts, and HALLUCANA in branch rollouts. Our proposal instead changes what the representation encodes at step $t$.
The constraints on design come from the bidirectional-modeling literature. Why Attention? proves that shallow combinations of forward and backward states cannot express some interactions between past and future (the "XOR limitation"). It shows that the fix has to happen inside the encoder, not in a layer stacked on top. We read this as a warning about the frozen-probe design used by DSVD and HALLUCANA: a shallow probe can only read out what the backbone already represents. Acquiring Bidirectionality via Large and Small Language Models gets bidirectional features by pairing a frozen forward LLM with a small backward LM. That works for tagging finished text but cannot run during generation, because the future text does not yet exist. It is therefore not a decoding-time competitor, though we use it as a detection-only reference. Older results on sentiment reversed by late clauses (Kratzwald et al.) illustrate the same point: early meaning often depends on late tokens.
The method draws on multi-task learning. CMHA formalizes learning from supervision that exists during training but is absent at inference. Future tokens are exactly this kind of supervision. RotoGrad separates negative transfer into mismatched gradient sizes and conflicting gradient directions. We reuse only its cheap magnitude-balancing step, since its rotation matrices do not scale to billion-parameter models. Faithful Finetuning is the closest multi-objective factuality fine-tuning work. It decomposes question answering into fact recall followed by answering and up-weights entity tokens. Its objectives stay strictly left-to-right, and it has no signal about how later text judges earlier tokens.
Research Program
Labels. We first build future-informed labels for text the model generates itself. For Llama-2-7B-Chat and Qwen-2.5-7B, we sample biographies and entity answers. We decompose them into atomic facts with the FActScore pipeline, check each fact against Wikipedia, and map each unsupported fact back to the token span that introduced it. We add question–answer data in the style of EntityQuestions (as in DSVD) and the span annotations of RAGTruth. Each span-initial position $t$ gets a hard label $y_t$: whether the span beginning there is later judged false with the whole text visible.
We also compute a soft label $\tilde y_t$. This is the prediction of a DSVD-style probe read at position $t+r$, which is the model's own delayed awareness. The target for the causal head at $t$ is a mix of the two, and we ablate the mixing. The evaluation topics used for FActScore are held out from this data.
Training. A small head $g$ reads the hidden state $h_t$ at a middle layer. We fine-tune with LoRA on the backbone plus the head, minimizing
$$ \mathcal{L} = \lambda_{\text{LM}}\,\mathcal{L}_{\text{LM}} + \lambda_{\text{F}}\,\mathrm{BCE}\big(g(h_t),\ \alpha y_t + (1-\alpha)\tilde y_t\big). $$
Here $\mathcal{L}_{\text{LM}}$ is next-token loss on the model's verified-correct generations mixed with general instruction data, and BCE is binary cross-entropy. The mixing weight $\alpha$ sets the balance between verified labels and delayed-probe labels. The weights $\lambda_{\text{LM}}$ and $\lambda_{\text{F}}$ are rebalanced every step so that the two losses' gradients on shared parameters have equal norm, following RotoGrad's magnitude step. The language-modeling loss matters because the auxiliary head would otherwise pull the representation away from fluent generation. Keeping gradient sizes matched is the cheapest available guard against that.
Decoding. Whenever the head's risk estimate $g(h_t)$ at a span start exceeds a threshold, we take the top $K$ candidate next tokens and run one batched forward step for each. We commit the candidate with the lowest predicted risk that stays within a probability margin of the greedy choice. If every candidate is high-risk, the model emits a hedge or abstains. This costs one step of lookahead, not HALLUCANA's $N$-step rollouts, and it acts before emission rather than after, unlike DSVD.
Analysis. The central measurement is an awareness-lag curve. For each hallucinated span starting at $t$, we measure how well a probe reading $h_{t+d}$ detects it, for offsets $d$ from $-3$ to $+5$, using AUROC (area under the ROC curve). We draw the curve for the base model and for the co-trained model. Our claim predicts that co-training moves the curve left: much higher detection at $d \le 0$, with little change at large $d$.
Baselines
The naive reference is the untuned model with greedy decoding. HALLUCANA reports a quality index $GQI_{0.2}$ of 0.68 for greedy decoding on FActScore, and DSVD reports Llama-2-7B-Chat at 31.9 Truth*Info on TruthfulQA. We rerun both numbers on our models, not copy them.
The strongest existing competitor is DSVD. It uses the same white-box access, runs on the same model families, and reports the most relevant numbers: Truth*Info on TruthfulQA from 31.9 to 48.4 for Llama-2-7B-Chat, and a 2.5-point FActScore gain for Qwen-2.5. We run it with its published probe-training recipe, but give its probes our label data, so the only remaining difference is the method.
HALLUCANA is the proactive-lookahead competitor, reporting $GQI_{0.2}$ of 1.69 on FActScore. We reproduce it on our models with its entropy trigger and a sweep over $K$ and $N$, again with classifiers trained on our labels.
Faithful Finetuning is the training-time competitor. It reports 51.41 MC1 and 74.00 MC2 on TruthfulQA, but only when combined with TruthX, so we compare against F2 alone at matched LoRA budget and data. The future-context detector (Lee et al.) is the detection-only reference. It reports sampled-future gains of up to 71.1 AUROC on SelfCheckGPT-style evaluation. We also include the backward-LM concatenation of Goto et al. as a detection-only reference on finished text.
All methods get the same base checkpoints and the same labeled data. Tuning uses the same number of hyperparameter trials on a shared development split. We report results at matched wall-clock latency per generated fact, measured with FActScore's tokens-per-fact accounting.
The key ablation isolates backbone co-training. It trains the identical head on the identical future-informed targets, but with the backbone frozen. A second ablation removes the future information by training the co-trained head on labels computed from the prefix only, which isolates the training-only future signal. A third replaces gradient-magnitude balancing with fixed weights.
To beat DSVD, the method must match or exceed its TruthfulQA and FActScore gains at equal or lower latency. It must also beat the frozen-head ablation on both detection at $d \le 0$ and end-task factuality.
Why not just…?
Why not just train a better probe on frozen hidden states with the same future-informed labels? This is the real threat, and the frozen-head ablation tests it directly. If a frozen probe reaches the same early-detection AUROC, then the information was already in $h_t$ and co-training the backbone is unnecessary. In that case the contribution shrinks to a labeling recipe for DSVD-style probes, and we would report it that way. The XOR result in Why Attention? and DSVD's own finding that detection improves only with delay both suggest that the early state is missing information, not merely hard to read. But that is a hypothesis, and the lag curve settles it.
Why not just roll back after the fact, as DSVD does? Rollback reacts only after an error has been emitted. It pays in regenerated tokens and in resampling that can drift toward a different error. If co-training moves detection earlier, prevention should reach DSVD's factuality with fewer discarded tokens. If DSVD's reported overhead of about 5% (and about 20% with multiple rollbacks) stays lower than ours at equal factuality, rollback wins. We measure both quantities directly.
Why not just sample futures or check consistency across drafts? Multi-draft checking of the SelfCheckGPT kind costs more than six times the compute of HALLUCANA, by HALLUCANA's account. It is also biased by snowballing: an early error propagates into the sampled futures, which is exactly what Lee et al. report. It also works only after generation. We keep it as a detection reference rather than a decoding competitor.
Why not just use a larger model? TruthfulQA finds that larger models are often less truthful: the best GPT-3 reached 58% against about 94% for humans. Goto et al. found that scaling a forward-only model did not close its gap on tasks that need right-hand context. We add one scale check by running the base 13B model against our co-trained 7B model.
Proposed Experiments and Decisive Checks
We evaluate on FActScore biographies (factual precision and $GQI_{0.2}$), TruthfulQA (Truth*Info, plus MC1 and MC2 for comparison with F2), and EntityQuestions, SciQ and StrategyQA following DSVD's protocol. For SciQ and StrategyQA we found no reported result on Lacuna, so we establish baselines ourselves. We use RAGTruth span-F1 to check detection quality. There, the strongest reported result is RL4HS-7B at 55.9 span-F1 from Learning to Reason for Hallucination Span Detection. Our head is not built for that task, so the check is only whether it lands in a reasonable range. To guard against losing general ability, we track perplexity on held-out general text and instruction-following quality.
The first decisive check is the awareness-lag curve. A leftward shift that the frozen-head ablation does not reproduce supports the core claim. No shift refutes it.
The second is end-task factuality at matched latency against DSVD and HALLUCANA. Here we must clear DSVD's reported TruthfulQA gain and HALLUCANA's reported FActScore $GQI_{0.2}$ on our reproductions.
The third check splits FActScore results by entity frequency. Entities the model has little knowledge of should show hedging and abstention, not recovered facts, because none of these methods can supply knowledge the weights lack. Gains on rare entities that come without more hedging would signal label leakage, not real improvement.
The fourth measures fluency and general-task loss with and without gradient balancing. This tells us whether negative transfer between the two objectives is real at this scale.
References
- DSVD: Dynamic Self-Verify Decoding for Faithful Generation in Large Language Models
- Enhancing Hallucination Detection via Future Context
- HALLUCANA: Fixing LLM Hallucination with A Canary Lookahead
- Why Attention? Analyze BiLSTM Deficiency and Its Remedies in the Case of NER
- Acquiring Bidirectionality via Large and Small Language Models
- Deep learning for affective computing: Text-based emotion recognition in decision support
- Assisting Main Task Learning by Heterogeneous Auxiliary Tasks with Applications to Skin Cancer Screening
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- Mitigating Large Language Model Hallucination with Faithful Finetuning
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- Learning to Reason for Hallucination Span Detection