Right-Context Support: Re-Reading a Model's Own Output to Catch Hallucinated Entities

A decoder-only language model writes each word seeing only what came before it. When it writes "Einstein was born in Ulm," the word "Ulm" is committed before the model has written anything that might confirm or contradict it. Hallucinations pile up exactly where this matters most. HalluEntity finds entity error rates rise from 9% at the start of a sentence to 36% at the end. Existing detectors judge each entity with the same left-to-right view under which it was produced, for example by looking at how unlikely or uncertain the token was. They pay for this in precision: HalluEntity reports that likelihood and entropy flag legitimate variation as hallucination.

We propose scoring every generated entity a second time, now with access to the text on both sides of it, and asking one question: does the text that follows make this entity more predictable than the text before it did? The tool is a known trick for giving causal models a view of later text, which is to repeat the input so that the second copy can attend to all of the first (Repetition Improves Language Model Embeddings, Re-Reading Improves Reasoning). We modify it in one way that makes the difference between a working and a useless signal: the entity is blanked out in the first copy. The result is a single-model, reference-free, sampling-free score for entity-level hallucination detection, plus a targeted repair step.

Core Claim

Let $L$ be the generated text to the left of an entity span $e$ and $R$ the text to its right. We define right-context support as

$$s(e) = \frac{1}{|e|}\Big[\log P(e \mid L, R) - \log P(e \mid L)\Big],$$

where $|e|$ is the span length in tokens and both probabilities come from the same frozen causal model. $P(e \mid L)$ is the ordinary left-to-right probability. $P(e \mid L, R)$ is estimated with an echo layout. The prompt first presents the passage with $e$ replaced by a blank, then asks the model to rewrite it, then replays $L$. We then score $e$ as the next tokens. Because the blanked first copy precedes the scored position, the model sees $R$ when it predicts $e$, but it cannot simply copy $e$.

Our claim is that factual entities are, on average, corroborated by what follows them: later dates, institutions, coreferent descriptions and co-occurring facts all point back to them. Fabricated entities are not corroborated. As a result, low $s(e)$ should separate hallucinated from correct entities with substantially higher precision than left-only uncertainty. The advantage should be largest for sentence-final entities and numbers, which HalluEntity reports error rates of 36.2% for, because those are exactly the positions where left-only scoring has the least information.

Why the Obvious Version Fails

The research file proposes a related idea: contrast an echoed second pass against an unechoed first pass during decoding. Taken literally, this breaks for detection. If the full output is repeated, the second copy of "Ulm" can attend to the first copy of "Ulm." Its probability then goes to nearly one through copying, regardless of truth. This is the same attention pattern that makes echo embeddings work for representations, and it destroys likelihood-based evidence.

During generation the problem is different but also fatal: every answer token already attends to the whole prompt. So repeating the prompt adds little information at exactly the tokens where hallucinations appear. What bidirectionality can actually add is right context of the output, and only after the output exists, and only if the scored span is hidden in the copy the model can see. We include the unmasked echo as an ablation, expecting it to collapse to chance-level separation. That result would show that masking is the essential ingredient and that naive "echo for factuality" schemes do not work.

Relation to Prior Work

Two papers motivate the proposal. HalluEntity supplies the end-of-sentence trap and the precision failure of uncertainty detectors. Factuality Enhanced Language Models for Open-Ended Text Generation shows that sentence endings are tightly constrained by factual entities. Its remedies, decaying the sampling threshold and adding topic prefixes in training, act during generation and never use the text that follows.

Three papers supply the method. Echo embeddings and RE2 show that repetition lets causal models attend backward without retraining. Both apply this to inputs, for embeddings and reasoning, not to scoring the model's own outputs. Echoes as Anchors shows that echoed prefixes anchor middle-layer attention in math reasoning. It also warns that echoing can entrench flawed paths through confirmation bias, which is precisely the copy collapse our masking is designed to prevent. RECALL uses two-step prompting to retrieve text that precedes a memorized cue, beating the reversal curse on 50 famous texts. Our infill scoring is a general, non-memorized version of the same move: we condition on what comes after in order to judge what comes before.

Do Language Models Know When They're Hallucinating References? is the closest competitor in spirit. It cross-examines the model with indirect questions over several samples and finds that fabrications show up as inconsistency. Our method replaces repeated sampling with deterministic scoring of the single text that was actually generated.

Decoding-time methods constrain the scope of what we expect. DoLa contrasts late against early layers. DeCoRe contrasts the full model against one with its context-copying attention heads disabled. Monitoring Decoding checks partial outputs with a larger model. All three judge tokens before the text that follows exists, and none can recover knowledge the model lacks. We share that limit and make no claim to inject new facts.

Research Program

The first component is the scorer itself. We will build it on Llama3-8B, the model whose HalluEntity results we compare against, and repeat it on a second open model family. Entity spans come from HalluEntity's annotations. Each span needs one extra forward pass, and the passes can be batched because the blanked passage changes only at the span. We will compare three ways of presenting the right context, to separate the signal from the prompt format: the blanked echo layout, an explicit fill-in-the-blank instruction, and a suffix-first ordering ("…R. Earlier in this passage: L"). The claim concerns the right-context signal; if the plain instruction does equally well, we will say so.

The second component is analysis of a confound. The right context was itself generated after the entity and conditioned on it, so a fabricated entity can be "supported" by text it caused. That is the autoregressive compounding described in the hallucination literature. We will measure this by stratifying entities according to whether they are referenced later in the text. We will also run a counterfactual control in which $R$ is regenerated from $L$ with the entity span left blank, so the right context cannot depend on the entity.

The third component is repair. For flagged entities, we resample $e$ from $P(\cdot \mid L, R)$ using the echo layout. Optionally we also regenerate the following sentence. We then measure whether factual error drops without the loss of diversity that factual-nucleus sampling incurs.

Baselines

The do-nothing reference is the trivial detector that predicts the base rate. In HalluEntity 15% of entities are hallucinated, so this sets the AUPRC floor near 0.15.

The left-only uncertainty baselines are mean token log-probability and entropy of each span under the same Llama3-8B scorer. These are exactly $\log P(e \mid L)$ and its entropy counterpart, so they isolate what $R$ adds.

The strongest existing reference-free detector in the file is HalluEntity's Focus method. On Llama3-8B it reports AUROC 0.78 and AUPRC 0.40 (HalluEntity). We will rerun it on the identical split rather than quoting it, to control for preprocessing. Beating it means higher AUPRC at matched or better AUROC, with a confidence interval obtained by bootstrapping over biographies, because precision is the documented weakness. As a target, an AUPRC gain of five points or more would count as meaningful.

The sampling competitor is the indirect-plus-direct query ensemble of Do Language Models Know When They're Hallucinating References?, adapted from citations to biography entities. The file gives no numerical AUC for it ("no reported result found on Lacuna for IQ+DQ AUC"). We will compare at two budgets: an equal number of forward passes per entity, and its own preferred sample count.

A self-verification prompt that asks the model whether the entity is correct given the full passage tests whether explicit judgment substitutes for scoring.

The key ablations are the unmasked echo, which should show the copy collapse, and the counterfactual-$R$ control. Together these isolate the masked right-context conditioning as the ingredient that works.

For repair on FactualityPrompts, the baselines are greedy decoding, factual-nucleus sampling and DoLa, all on the same model with equal hyperparameter search budgets. The original FactualityPrompts paper reports named-entity error falling from 33.3% to 14.5% and entailment rising from 11.8% to 25.5%. Those numbers come from a 530B model with training changes, so they serve as context rather than a target.

Why not just…?

Why not just use token entropy or likelihood, which is free? HalluEntity shows these signals have high recall but low precision, because a correct but rare name is also uncertain. Our score subtracts exactly that left-only term. If $s(e)$ only matches the left-only baselines on AUPRC, the proposal fails on its central claim.

Why not just sample several responses and check consistency? This is a real threat on accuracy. Monitoring Decoding notes that sampling still fails when the model is confidently wrong, and that fine-grained self-consistency costs 316.72 ms per token. Sampling may still beat us when budget is unlimited. The matched-forward-pass comparison settles this. We also test whether the two signals are complementary by combining them, which would be a useful result even if neither dominates.

Why not just ask the model whether each entity is correct? This uses the same right context with no special layout, and nothing in the file measures it for entity detection. We run it directly. If it matches $s(e)$, the contribution narrows to the finding that right context is what helps, and we will report that.

Why not prevent hallucinations during decoding with DoLa or factual-nucleus sampling? These methods act before any right context exists, which is the regime of the end-of-sentence trap. Our repair stage is evaluated on FactualityPrompts against both. Detection can also be stacked on top of either method, so we will additionally test DoLa followed by our repair.

Decisive Checks

The main experiment scores HalluEntity's 18,785 annotated entities and reports AUROC and AUPRC overall and broken down by sentence position and entity type. Our prediction is that the gain over left-only scoring grows toward sentence-final positions and numeric entities. If the gain is flat across positions, the right-context explanation is wrong even if the aggregate number improves.

Second, the unmasked echo should fall to near-random AUPRC. Third, under the counterfactual-$R$ control, support for hallucinated entities should drop while support for correct entities holds. That would show the signal comes from corroboration and not from text the entity caused.

As a transfer test, we apply the scorer to fabricated citations in the ACM reference setting of Do Language Models Know When They're Hallucinating References?, where GPT-3 fabricates 73.6% of titles. Here we score titles conditioned on generated author and venue fields. We expect little benefit on short-answer TruthfulQA, where there is almost no right context. This mirrors the small gains echo embeddings report for inputs under about 64 tokens, and we report it as a stated boundary of the method.