When a language model writes a biography and states that its subject "was born in Ohio in 1952," a true answer and a made-up one can look the same to standard detectors. Both are specific. Both are somewhat surprising to the model, since many birthplaces are plausible. Both are fluent. This proposal tests a different signal. A genuine fact about a person should depend on which person the text is about. A fabricated one is often a generic slot-filler that the model would have written for almost anyone. The test needs no gradients and no external knowledge. We replace the entity the discourse is centered on with other names of the same type, rerun the model forward, and measure how much the probability of the generated fact changes. A fact that barely moves when the subject is swapped is a hallucination suspect.
The idea comes from Centering Theory, a model of local coherence in which each sentence has a "center," the entity the passage is currently about, and coherent text keeps its content tied to that center from one sentence to the next. Coherence research has used this for decades to score text quality. It has only been linked to factuality through reference-based evaluation. We turn it into a reference-free, token-level hallucination detector for open-ended generation. We then check whether it explains a striking finding: entity hallucination rates rise from 9% at the start of a sentence to 36% at the end (HalluEntity).
Core Claim
For an entity span $e$ generated after prefix $x$ whose current discourse center is $c$, define center sensitivity as
$$S(e) = \log p(e \mid x) - \frac{1}{K}\sum_{k=1}^{K} \log p(e \mid x_{c \to c'_k}),$$
where $x_{c \to c'_k}$ is the prefix with every mention of $c$ replaced by a same-type substitute $c'_k$, and $K$ is the number of substitutes. We claim that low $S(e)$ identifies hallucinated entities with much better precision than entropy or likelihood. The reason is that $S(e)$ separates two things raw uncertainty mixes together. One is legitimate uncertainty about a specific fact. The other is a model that has stopped conditioning on its subject. We further claim that the end-of-sentence trap reflects a within-sentence decay of $S$: the model's dependence on the center weakens as a sentence moves on to new information.
This differs from the closest existing work in three ways. DiscoScore tracks Centering-style entity foci between adjacent sentences and shows that discourse coherence correlates with human factual-consistency ratings. It compares a hypothesis against a gold reference and scores whole texts, whereas we need no reference and localize individual entities. Co-opNet reranks complete candidate summaries with a sentence-adjacency discriminator and finds that this improves factuality. That establishes local coherence as a factuality proxy, but the signal is a separately trained classifier over whole candidates. Ours is a counterfactual read-out of the generator itself at each entity. Hallucination Begins Where Saliency Drops shows in vision-language models that hallucinated tokens follow a collapse in gradient-based dependence on earlier output. Our hypothesis is a text-side, discourse-specific version of that finding: the dependence that matters is on the center, not on a recency window. We measure it with forward passes only, which avoids the backward passes behind SGRS's reported "roughly 30–40% latency overhead" and its failure to run at 72B scale.
Research Program
The first component is the detector. For each generated sentence we identify the center using standard Centering heuristics: the highest-ranked entity carried over from the previous sentence, which defaults to the subject in biographies. We draw substitutes from two pools. The first is real people of matched type and era, whose true facts differ. The second is fictional names, which reveal the model's generic prior. We keep these pools separate because they answer different questions. Real substitutes test whether the fact is person-specific. Fictional substitutes test whether the fact is simply the default answer. The $K$ substituted prefixes run as one forward batch, and scores for every entity in a sentence come from the same batch. The expected cost is therefore $K$ extra prefill passes per sentence rather than per token.
The second component is analysis. Using HalluEntity's 18,785 entity labels, we fit hallucination probability against sentence position with and without $S(e)$ and discourse status as covariates. Discourse status means whether the entity is given (already mentioned) or new, and which Centering transition type it enters on. If the positional effect largely disappears once $S$ is included, the trap is a center-dependence phenomenon. If it persists, compounding errors within the sentence are a separate cause. We also ask whether $S$ declines with position for truthful entities, which would indicate that the drift is systematic rather than specific to errors.
The third component is a modest mitigation step. Whenever a sentence-final entity falls below a calibrated $S$ threshold, we resample only that span, with the prefix left unchanged. If no resample passes, we emit a hedge. This intervenes during generation, which addresses the gap Co-opNet leaves open when every pooled candidate is already wrong.
The fourth component tests transfer to grounded summarization, where the center is a source-document entity. On FRANK we test whether low $S$, now computed by swapping the source antecedent, flags the coreference and discourse-link errors that FRANK finds persist even in large pretrained summarizers.
Baselines
The naive reference point is raw token likelihood and entropy over each entity span, computed from the same Llama3-8B generation pass. HalluEntity reports that these methods have high recall but poor precision, so they set the floor. With a 15% base hallucination rate, a random detector has an area under the precision-recall curve (AUPRC) near 0.15.
The strongest existing detector is Focus, as evaluated in HalluEntity. Focus propagates uncertainty through keywords and reports an area under the ROC curve (AUROC) of 0.78 and an AUPRC of 0.40 on Llama3-8B. It is the right competitor because it was built for the same entity-level precision problem on the same labels. We will rerun Focus from its published description on identical entity spans and the identical model rather than copy numbers. Beating it means a higher AUPRC with a bootstrap confidence interval that excludes Focus's. AUPRC is the primary target because precision is where uncertainty methods fail.
Because our score costs extra forward passes, we also include a compute-matched sampling-consistency baseline. It draws $K$ regenerations of each sentence and flags entities that do not recur, in the spirit of the indirect-query consistency checks in Do Language Models Know When They're Hallucinating References?. Its $K$ is set so that total token computation equals ours. We also report prompted self-detection, for which HaluEval gives ChatGPT's summarization self-detection accuracy as 58.53% (HaluEval).
Two ablations isolate the key ingredient. A recency ablation deletes the previous sentence instead of swapping the center, which is the Markovian local-context signal of classic sentence-ordering models and the textual analogue of LocoRE's recent-window focus. A non-center ablation swaps a same-type entity that is not the center. If either ablation matches full center sensitivity, the Centering component adds nothing.
For mitigation, the baselines are unmodified decoding, entropy-triggered span resampling at the same trigger rate, and a text adaptation of LocoRE, which boosts attention to recent tokens. We found no reported result on Lacuna for any of these on text biography generation, so all of them will be run fresh.
Why not just…?
Why not just threshold entropy or likelihood? HalluEntity already shows why this fails. True informative entities have high entropy because many phrasings are valid, which ruins precision. Our score subtracts a counterfactual log-probability, which should cancel variation driven by phrasing that is shared across substitutes. The head-to-head AUPRC comparison tests whether it does.
Why not just calibrate uncertainty by position and entity type? This is a real threat. The end-of-sentence trap and the high hallucination rates for proper nouns and numbers (30.9% and 36.2% in HalluEntity) mean that a logistic regression on entropy, position, and type could absorb most of any gain. We therefore fit exactly that model and report whether $S$ adds significant AUPRC on top of it. If it does not, the claim fails, and the paper becomes a negative result about the trap.
Why not just sample several times and check consistency? Self-consistency is strong and needs no entity typing, so we match its compute. The two methods should fail in different places. A model can consistently produce the same generic default, such as a common nationality, across samples. That default is stable under resampling but insensitive to swapping the center, and our score should catch it. We will test this specifically on entities where the two methods disagree.
Why not just ask the model? HaluEval's 58.53% self-detection accuracy in summarization, and ChatGPT's 61.52% against InsertGNN's 71.43% on local sentence insertion (InsertGNN), suggest that prompting captures local discourse dependence poorly. We include it only as a reference point.
Proposed Experiments
The main experiment regenerates HalluEntity-style biographies with Llama3-8B, uses HalluEntity's FActScore-derived entity labels, and reports AUROC and AUPRC against all baselines at matched compute. It also reports wall-clock overhead, which we expect to stay well below SGRS's 30–40% because no backward pass is needed.
The first decisive check is the ablation ordering. Center-swap sensitivity must beat both the recency ablation and the non-center ablation. Otherwise the discourse framing is decoration on generic context sensitivity.
The second decisive check is the regression on the positional trap. If adding $S$ as a covariate does not substantially shrink the 9%-to-36% positional gradient, our mechanistic story is wrong even if detection improves.
A known failure mode is facts that are true but generic, such as common nationalities, which will have low $S$ and lower precision. We report precision separately for the most frequent entity values so that this cost is visible.
For transfer, we compute $S$ on FRANK summaries from CNN/DM and XSum by swapping source antecedents. We report the per-summary Pearson correlation with human factuality judgments against FRANK's reported ceiling for existing metrics of max Pearson $\rho \approx 0.20$, and we break results down by FRANK's discourse-error categories. For mitigation, we report FActScore-style precision and the fraction of hedged spans against the decoding baselines.
References
- HalluEntity: Benchmarking and Understanding Entity-Level Hallucination Detection
- DiscoScore: Evaluating Text Generation with BERT and Discourse Coherence
- Discourse Understanding and Factual Consistency in Abstractive Summarization
- Hallucination Begins Where Saliency Drops
- Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
- Do Language Models Know When They're Hallucinating References?
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- InsertGNN: A Hierarchical Graph Neural Network for the TOEFL Sentence Insertion Problem