When large language models (LLMs) generate text, they rarely fail completely. Instead, they produce outputs that are mostly fluent and accurate, interspersed with isolated factual errors or ungrounded claims. Traditional evaluation metrics treat hallucination detection as a binary or sentence-level classification problem. However, this coarse approach acts as a blunt instrument, failing to capture the nuance of partially correct generations.
To address this, a growing body of research focuses on span-level hallucination detection—the practice of pinpointing the exact word- or phrase-level boundaries of unfaithful text. By localizing errors surgically, researchers can diagnose model failures more accurately and enable targeted, context-preserving corrections.
The Shift from Global to Local Evaluation
Evaluating long-form text generated by AI is notoriously difficult. As Yao Dou et al. observe in Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text, traditional global scores fail to provide a granular analysis of model failures, such as isolating specific logical contradictions or factual inaccuracies.
The core issue is that hallucinations are frequently "partial." As Weijia Xu et al. point out in Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection, standard detection techniques prioritize detached hallucinations where an entire sequence is irrelevant. Identifying partial hallucinations within a largely correct output remains a distinct and necessary challenge. When an evaluator evaluates a document as a whole, fine-grained errors are often averaged out or missed entirely.
To bridge this gap, frameworks like FALTE, introduced by Tanya Goyal et al. in FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation, shift the paradigm by prompting annotators to identify specific text spans containing errors. This granular approach transforms evaluation from a simple quality score into a precise diagnostic map.
Mechanisms for Pinpointing Error Boundaries
How can an automated system or an external evaluator locate the exact boundaries of a hallucination? Current literature approaches this through three primary mechanisms: model introspection, token-level sequence modeling, and structured prompting.
1. Model Introspection
Models often exhibit internal behavioral shifts when they begin to hallucinate. Weijia Xu et al. discovered that hallucinating translation models tend to "get stuck" on a small subset of input tokens, distributing attention abnormally. Similarly, Sky CH-Wang et al. demonstrate in Do Androids Know They're Only Dreaming of Electric Sheep? that a transformer's internal hidden states contain latent indicators of truthfulness. By training simple classifiers (probes) on these hidden states, they could detect specific ungrounded spans of text within a longer sequence, showing that the model's internal representations "know" when a generation has gone off the rails.
2. Token-Level Sequence Modeling
Instead of treating detection as a generation task, some approaches frame it as a token-level sequence labeling problem. In FunghiFunghi at SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes, Tariq Ballout et al. adapted SpanBERT—a model natively designed for span-based predictions—to successfully pinpoint the boundaries of factual inaccuracies across multiple languages.
3. Structured Prompting and Delimiters
When using LLMs themselves as judges, the method of extraction matters. Xu Liu et al. note in CCNU at SemEval-2025 Task 3: Leveraging Internal and External Knowledge of Large Language Models for Multilingual Hallucination Annotation that instructing LLMs to output numerical indices for errors often yields poor results due to extraction errors. Instead, instructing the model to directly mark hallucinated text within the content using specific delimiters (like brackets or tags) significantly improves boundary precision.
Why It Works: The Power of Actionable Feedback
The primary advantage of span-level detection over sentence-level detection is that it provides actionable feedback for model refinement. If a model is told an entire sentence is wrong, it may inadvertently alter correct information or introduce new errors during revision.
Span-level localization acts as a precise pointer. In Learning to Refine with Fine-Grained Natural Language Feedback, Xinyu Zhao demonstrates that localized feedback serves as an anchor, guiding the model to address only the identified factual errors while preserving the accurate surrounding context.
Sentence-Level Critique (Blunt):
"The sentence 'Apples are blue and grow on trees' is factually incorrect."
-> Result: Model might rewrite the whole sentence, losing the correct context that apples grow on trees.
Span-Level Critique (Precise):
"In the sentence 'Apples are [blue] and grow on trees', the span 'blue' is incorrect."
-> Result: Model targets only the error, correcting it to 'red' or 'green' while preserving the rest.
This concept is operationalized by Rachneet Sachdeva et al. in Localizing and Mitigating Errors in Long-form Question Answering. Through a method called Error-Informed Refinement (EIR), a feedback model identifies specific sentence-level omissions and span-level inaccuracies, passing these targeted signals to an LLM to prompt surgical, human-preferred corrections.
Standardizing the Field: Benchmarks and Toolkits
As the community recognizes the necessity of fine-grained evaluation, several benchmarks and frameworks have emerged to standardize how span-level errors are tracked and measured:
- factgenie: Developed by Zdeněk Kasner et al., this framework provides a unified interface for word-level span annotation, allowing both humans and LLMs to assign discrete error categories to specific text segments.
- SNaC: Introduced by Tanya Goyal et al., this framework focuses on fine-tuned sequence-to-sequence models that pinpoint spans causing narrative incoherence in long-form summaries.
- Mu-SHROOM: Organized by Raul Vazquez et al. in SemEval-2025 Task 3: Mu-SHROOM, the Multilingual Shared-task on Hallucinations and Related Observable Overgeneration Mistakes, this shared task standardizes multilingual evaluation by forcing models to identify specific content words that lack support from reference documents across 14 languages.
Despite these advancements, challenges remain. As Flor Alberts et al. note in FENJI at SemEval-2025 Task 3: Retrieval-Augmented Generation and Hallucination Span Detection, automated systems still struggle with overlapping error spans and minor textual variations. However, by treating hallucinations as localized anomalies rather than systemic failures, span-level detection provides the surgical precision required to make text generation models truly reliable.
Key Foundational Work
FALTE: A Toolkit for Fine-grained Annotation for Long Text Evaluation (2022) — contributed 1 concept elements to this direction.
Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Text (2022) — contributed 1 concept elements to this direction.
Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection (2023) — contributed 1 concept elements to this direction.