A decoder-only language model, the GPT/LLaMA/Qwen kind that writes one token at a time and lets each position see only the tokens before it, has a structural blind spot when it hallucinates. At the moment it writes a subject such as "Marie Curie", its internal state cannot yet reflect the claim that will follow, "…won the Nobel Prize in Chemistry in 1903". By the time the wrong year appears, the error is already part of the model's context, and later tokens build on it. Encoder models like BERT avoid this because every token attends to every other token. This note proposes to borrow that property only for checking. After the model finishes a sentence, the same frozen model re-reads it once with full two-way attention inside the sentence, using a small trained module. It labels which tokens are likely fabricated. Generation then rolls back to just before the first flagged token and resamples. The generator's own forward pass is never changed.
The bet is narrow and falsifiable. Two-way re-reading should not help much with deciding whether a sentence is wrong, because the last token of a causal pass already sees the whole sentence. It should help with deciding where the error is, because the causal state at an early token cannot see the later tokens that expose it. Locating the error is what makes a minimal rollback possible: replace the bad year rather than regenerate the paragraph. If that localization advantage does not appear, the idea reduces to an expensive probe, and the experiments are designed to show that clearly.
What the prior work establishes and leaves open
The motivating evidence comes from UBMoE-LLM. It adds bidirectional feed-forward experts beside the frozen causal ones in Qwen1.5 and reports, unexpectedly, that TruthfulQA scores improved. The research file reports no number for that improvement. The same paper also constrains the design. It shows that fine-tuning a causal model with two-way attention destroys generation, because token representations start to depend on future tokens that do not exist at inference time; the paper calls this "subsequent dependence". Our design sidesteps this by construction. The two-way pass is a separate reading mode whose outputs are never fed back into generation, so the generator cannot learn to rely on the future.
Two existing pieces supply the method. The shared-cache idea of ICaRus treats a frozen decoder as a "logical encoder" whose key-value cache, the stored attention inputs for past tokens, is reused by lightweight LoRA-adapted heads. The re-reader likewise reuses the generator's cache for everything before the current sentence and recomputes only the sentence's own tokens with a two-way mask. ICaRus never studied factuality, and we use its machinery for a different purpose. Dense token-level supervision for the reader follows the spirit of side-network adaptation in LST. The backbone is never backpropagated through for the generator, though here the trainable part is a LoRA on attention within the re-read block plus a token classifier.
Most of the hallucination direction consists of causal-side detectors, which are our main competitors. Attention Head Entropy trains a sparse logistic regression on how spread out attention is within each head. It reaches 0.88 average AUROC for answer correctness across TriviaQA, HotpotQA and MedMCQA, and 0.74 using question tokens alone. Activation Steering Decoding finds that hallucinated and factual activations are linearly separable with 82.5% accuracy in middle-to-late layers of vision-language models. Both score whole answers, or tokens using only leftward context. Neither asks whether that leftward view limits localization. Attribution-Guided Decoding steers tokens at high-entropy forks, but it can only choose among the model's own top candidates and cannot revisit tokens already committed. Our rollback is aimed at exactly that committed-error cascade.
External verifiers pursue a parallel line. Zero-Shot Multi-task Hallucination Detection uses a DeBERTa-v3 entailment model on SHROOM and reaches 0.78 accuracy model-agnostic. OPDAI reaches 0.836. Co-opNet reranks GPT-2 outputs with bidirectional discriminators. These verifiers are bidirectional, but they are separate models that lack the generator's parametric knowledge. Their natural setting is source-grounded checking, not closed-book recall. Finally, BRIDO uses agreement among sampled candidates to fight hallucination in encoder-decoder summarizers. Its consensus logic is the basis of the compute-matched sampling baseline below.
Core Claim
A frozen causal LM, given a small trained module that re-reads its own most recent sentence with intra-sentence bidirectional attention, localizes fabricated tokens substantially better than any causal-only probe on the same model. The claim is specifically about span-level localization. Rolling back to the first flagged token reduces closed-book hallucination more, per unit of extra compute, than sentence-level rejection or best-of-K sampling. At sentence-level detection, we expect parity with a last-token causal probe. Parity there is a prediction of the hypothesis, not a failure of it.
Research Program
The re-reader is built in three parts. The first is the reading pass. When the generator emits a sentence boundary, we take its cached keys and values for all prior context. We then recompute the sentence tokens with a mask that lets them attend to each other in both directions and to the cached prefix causally. A rank-16 LoRA on the attention projections is active only in this pass. The second part is a per-token classifier. It is a linear head on a concatenation of middle and late layer states, chosen because both the head-entropy and steering papers locate the signal there. It outputs a probability that each token belongs to a fabricated span. The third part is the rollback policy. If any token exceeds a threshold, generation truncates to just before the earliest flagged span, bans the previously chosen token at that position, and resamples. This repeats for at most two retries per sentence. The extra cost is one forward pass over the sentence tokens, plus any regenerated tokens.
Token-level supervision does not exist in the benchmarks we use, so we construct it in two ways. The first source is HaluEval's QA and dialogue subsets in HaluEval, which pair correct and hallucinated responses to the same query. Aligning the two strings labels the differing spans. The second source is the model's own closed-book answers on TriviaQA and HotpotQA. We label spans by matching against gold answers and supporting facts, and add controlled entity and relation swaps. Those swaps target the "role scrambling" failure mode, where the evidence of error comes after the entity it implicates. Training data and labels are identical across the re-reader and every probe baseline.
We run on the Qwen1.5 family at 1.8B, 7B and 14B, to stay comparable with UBMoE-LLM, and on LLaMA-3.1-8B, to stay comparable with ICaRus and AGD. The analysis will report detection quality as a function of where the error-revealing token sits relative to the error. That is the direct test of the causal-blind-spot mechanism.
Baselines
The naive reference point is the unmodified model with greedy or nucleus decoding and no verification. It anchors every generation metric. On TruthfulQA, the original paper TruthfulQA reports 58% truthfulness for its best GPT-3 model against 94% for humans. That gap motivates the work but is not a target for our model sizes.
The strongest internal competitor is the head-entropy probe of Attention Head Entropy, whose reported number is 0.88 average AUROC for answer correctness across TriviaQA, HotpotQA and MedMCQA. We pair it with a linear probe on causal hidden states in the style of Activation Steering Decoding, applied both at the sentence's last token and per token. Both are the strongest competitors because they use the same model's internals at near-zero cost. Any gain from re-reading must exceed what they already extract. We retrain both on our exact data, layers and label sets, and give them the same hyperparameter search budget as the re-reader. That budget is twenty configurations each, selected on a held-out split. For sentence-level detection we expect a tie with the last-token probe. Beating these probes means a clear improvement in span-level F1 and in AUROC computed per token. No reported result was found on Lacuna for token-level localization by either method, so we establish those numbers ourselves.
The strongest external competitor is an entailment-based verifier. We use DeBERTa-v3 as in Zero-Shot Multi-task Hallucination Detection, which reports 0.78 model-agnostic accuracy on SHROOM, while OPDAI reports 0.836. On closed-book tasks we give it retrieved evidence where available, and we also fine-tune it on our span labels with the same budget. On HaluEval, the reference detection numbers are ChatGPT's 62.59% on QA and 58.53% on summarization.
For generation-time mitigation, the competitor is best-of-K sampling with consensus selection. This is BRIDO's agreement idea moved from training to inference, with agreement measured by entailment between samples. K is set so that total generated tokens match the re-reader's average cost. We also include whole-sentence rejection driven by the last-token causal probe.
The key ablation keeps everything about the re-reader identical, including the extra pass, the LoRA, the data and the classifier, but uses a causal mask. This isolates two-way attention from the confounds of extra compute and extra trainable parameters. If the causal-mask re-reader matches the bidirectional one, the central claim is false.
Why not just…?
Why not just probe the last token of the sentence, which already sees the whole sentence causally? This is the most serious objection, and for sentence-level detection we concede it in advance. The last token has seen everything the two-way reader sees. Our claim rests on localization, where an early token's causal state structurally lacks the later evidence. The position-of-evidence analysis and the causal-mask ablation decide this directly. If per-token causal probes, perhaps combined with the last-token score, localize as well, we will report that the idea does not survive.
Why not just use an external entailment model like DeBERTa? It is already bidirectional, cheap, and strong on source-grounded checking, with 0.78–0.836 on SHROOM. We expect it to remain competitive, and possibly to win, on SHROOM-style tasks where the source text is given. Its weakness is closed-book factual recall, where nothing external is available to entail against, and the generator's own parameters hold the relevant knowledge. We report the two regimes separately rather than averaging them, so a split verdict will be visible.
Why not just sample several answers and keep the consensus? Agreement among samples does reduce hallucination, as BRIDO shows for summarizers. It is also the right baseline at matched compute. It shares the candidate bottleneck noted for AGD and Co-opNet, though: if the model's samples agree on a misconception, as TruthfulQA's inverse-scaling result suggests larger models often will, consensus confirms the error. Localized rollback with a banned token forces a different completion at exactly the suspect position. Whether that helps more than K samples at equal token cost is the main generation experiment.
Why not just ask the model, or a stronger LLM, to critique its answer? HaluEval shows ChatGPT detects hallucinations in summaries at 58.53% accuracy, little better than chance, and that showing it contrasting candidates made detection worse. We include self-critique prompting as a cheap reference. The prior evidence makes it unlikely to be the main threat.
Proposed Experiments
The first decisive check is localization. On span-labeled HaluEval QA and dialogue and on our TriviaQA and HotpotQA swap sets, we compare per-token AUROC and span F1 for the bidirectional re-reader, the causal-mask re-reader, per-token causal and head-entropy probes, and fine-tuned DeBERTa. Results are stratified by the distance between the error token and the token that reveals it. The prediction is that the bidirectional advantage grows with that distance and vanishes when the revealing token comes first. The second check is sentence-level detection on HaluEval and SHROOM, reported against the cited 62.59%, 58.53%, 0.78 and 0.836 figures, where we expect parity rather than gains over the last-token probe.
The third check is generation. We measure TruthfulQA truthfulness and informativeness, closed-book exact match and fabricated-span rate on TriviaQA and HotpotQA, and HaluEval QA response accuracy. Each method is plotted against tokens of extra compute: rollback, sentence rejection, compute-matched best-of-K consensus, and no intervention. A fourth check confirms the generator is untouched. MMLU accuracy should be unchanged on Qwen1.5-1.8B, against the 48.2 average UBMoE-LLM reports preserving, because the reader never writes into the generator's cache. Scale trends across 1.8B to 14B will show whether the causal blind spot narrows as models grow.
The idea is supported only if the bidirectional reader beats its own causal-mask ablation on localization and that advantage turns into better hallucination reduction per unit of compute than best-of-K. If either fails, the note's conclusion will say so.
References
- What Limits Bidirectional Model's Generative Capabilities? A Uni-Bi-Directional Mixture-of-Expert Method For Bidirectional Fine-tuning
- ICaRus: Identical Cache Reuse for Efficient Multi-Model Inference
- LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning
- Attention Head Entropy of LLMs Predicts Answer Correctness
- Activation Steering Decoding: Mitigating Hallucination in Large Vision-Language Models through Bidirectional Hidden State Intervention
- Attribution-Guided Decoding
- Zero-Shot Multi-task Hallucination Detection
- OPDAI at SemEval-2024 Task 6: Small LLMs can Accelerate Hallucination Detection with Weakly Supervised Data
- Cooperative Generator-Discriminator Networks for Abstractive Summarization with Narrative Flow
- BRIDO: Bringing Democratic Order to Abstractive Summarization
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods