Is the Span-Level Hallucination Detection Leaderboard Measuring Boundaries or Detection?

Systems that flag hallucinations in model-written text are increasingly scored at the span level. A detector must mark the exact stretch of words that the source document does not support, and it is graded by the overlap between its marks and a human annotator's marks. This grading has a hidden problem. A detector can find the right false claim and still lose most of its credit because it marked "in 1992 the company" where the annotator marked "1992". Two humans often disagree about exactly such edges. The source passages may also have been seen in pretraining. A single span-overlap score therefore blends three different things: did the detector find the unsupported claim, did it draw edges the way this benchmark's annotators happen to, and was the material familiar to it.

We propose to separate these three quantities. We measure how much human annotators disagree on hallucination span boundaries in existing benchmarks. We build a fresh test set where hallucinations are inserted on purpose, so their boundaries are known exactly and the text cannot have been memorized. We then re-score current detectors with a metric that reports localization and boundary placement separately. The practical question is whether recent leaderboard gains on RAGTruth come from better detection or from learning annotator boundary conventions. The answer decides whether the field should invest in reasoning-based training such as reinforcement learning, or in cheap boundary correction.

Core Claim

On RAGTruth, a specialized 7B detector trained with reinforcement learning reaches 55.9 span F1. Supervised fine-tuning reaches 50.1 and the much larger o3 reaches 51.2 (Learning to Reason for Hallucination Span Detection). Earlier, a fine-tuned Llama-2-13B scored 52.7 against 28.3 for prompted GPT-4 (RAGTruth).

We claim that a substantial part of the advantage of detectors trained on RAGTruth comes from boundary placement, not from finding hallucinations. If that is true, three things follow. A small boundary-only corrector applied to an untrained model's outputs should recover much of the gap. On inserted hallucinations with exact boundaries, the ranking should compress or reorder. And the gaps between the top systems should fall inside the disagreement between two human annotators.

If the claim is false, the gaps should survive all three tests. That outcome would also be informative, because it would mean the reinforcement-learning gains reflect genuine detection ability.

Research Program

Prior work plays three roles here, and the program draws on each.

The work that motivates the program shows that benchmark noise hides real model differences. The audit of CoNLL 2003 in Named Entity Recognition - Is There a Glass Ceiling? traced shared model errors to dataset errors, including span boundaries, and found that FLAIR's F1 rose from 93.09 to 93.79 once these were fixed. Re-TACRED found that 23.9% of TACRED labels were wrong and showed that noise made PA-LSTM and C-GCN look identical. Neither study touched generated text or hallucination spans.

The work that supplies the method is the boundary-correction line. Correct after Answer classifies candidate spans as correct, partially correct, or wrong, and then uses a pointer network to move the edges of the partially correct ones. Boundary repositioning with a pointer network re-reads a small window around a rough span to choose new start and end tokens. We reuse this design as a measuring instrument rather than as a product.

The work that constrains the design is Back to Square One. It shows that models solve benchmarks through surface artifacts, so any synthetic test set we build must be checked for giveaways. Auditing the Audit warns that audit pipelines themselves fail silently, which is why we report sanity controls alongside every result.

Component 1: annotator boundary audit. We double-annotate about 300 RAGTruth test responses with fresh annotators who follow RAGTruth's own guidelines. From these we compute human-versus-human span F1 and the distribution of start and end disagreements, broken down by RAGTruth's four hallucination categories. We expect "subtle" baseless information to have the fuzziest edges. Mu-SHROOM (Hallucination Detectives at SemEval-2025 Task 3) ships soft labels from multiple annotators. This gives a second, free estimate of boundary disagreement for English, Spanish, French and Arabic.

Component 2: a split metric. Each predicted span is matched one-to-one to a gold span whenever the two overlap at all. This matching yields localization F1, which asks whether the detector found the right claims. For matched pairs, we report the mean absolute start and end offsets in tokens, written $|\Delta_s|$ and $|\Delta_e|$. These measure how far the predicted edges sit from the annotator's edges. Standard span F1 is still reported. Our hypothesis says that systems differ more in offsets than in localization F1.

Component 3: a fresh insertion benchmark. Following the logic of DyVal, we generate the test set rather than collect it. DyVal does this for symbolic reasoning, and the research file notes it cannot produce text extraction tasks. The pipeline starts from source passages published after the training cutoffs of the evaluated models, namely news, Q&A pages, and reviews. A model writes a faithful response, which humans verify on a sample. A second step then rewrites exactly one constituent so that it contradicts the source or adds unsupported content. The rewrites are stratified by RAGTruth's taxonomy: evident conflict, subtle conflict, evident baseless info, and subtle baseless info. Because the rewrite is a known edit, the gold boundary is exact by construction. Because the source is new, memorization cannot help. Hazra-style or cutoff-based reasoning (Unveiling Causal Reasoning in Large Language Models) motivates the post-cutoff choice.

Component 4: a boundary-only corrector. We train an edge-moving pointer network, in the style of the ACC corrector, on RAGTruth training pairs of predicted and gold spans. It may shift a detected span's edges within a small window but can never add or delete a span. This restriction is what lets it isolate the boundary effect.

Baselines

The naive reference points are an empty predictor and a whole-response predictor. The empty predictor shows the degenerate high-precision corner that RL4HS warns about. The whole-response predictor shows how much credit pure overlap gives. The untrained Qwen2.5-7B, which scores 16.2 span F1 in that paper, anchors the low end among real models.

The strongest existing method is RL4HS-7B, reported at 55.9 span F1 on RAGTruth. Its closest competitors are the supervised fine-tuned model at 50.1 and o3 at 51.2, both from Learning to Reason for Hallucination Span Detection. We will retrain the SFT and RL variants from the same Qwen2.5-7B starting point on the same RAGTruth training split, with an equal tuning budget measured in GPU-hours and validation sweeps. We will query o3 or the best available reasoning model with a single fixed prompt. All systems are scored by one evaluation script, and we first reproduce the reported numbers within one point before trusting the script.

The key ablation adds the boundary-only corrector to each system's outputs. Its effect is measured as how much of the gap between the prompted model and RL4HS it closes. A second ablation trains the corrector on exact synthetic boundaries instead of RAGTruth annotator boundaries. If this version helps on the insertion set but hurts on RAGTruth, it shows that "correct" edges and "annotator" edges differ.

On the insertion benchmark, a method beats RL4HS on detection if its localization F1 exceeds RL4HS's by more than the confidence interval from a paired bootstrap over items. The "boundary convention" hypothesis is supported if the corrector closes at least half of the RAGTruth gap between the prompted model and RL4HS while closing little of it on the insertion set.

Why not just…?

Why not just use a lenient overlap metric and call it done? Token-overlap or any-overlap scoring would hide boundary differences. It cannot tell whether a residual gap reflects missing claims or a predicted span that merely grazes a long gold span. It also says nothing about whether the gold edges are themselves consistent. We still include a lenient metric as a reported baseline. If the lenient metric and our localization F1 rank systems identically everywhere, the split metric adds little, and we will say so. The offset statistics and the human ceiling would remain as the contribution.

Why not just re-annotate the RAGTruth test set, as Re-TACRED did for TACRED? Full re-annotation is expensive and would remove one annotator's conventions only to install another's. The research file flags exactly this risk for ambiguous cases. It also does nothing about familiarity with the data. Our double-annotated subset delivers the part of re-annotation we need, which is an estimate of the human ceiling. The insertion set avoids the subjectivity problem entirely, because its boundaries come from known edits.

Why not just snap predictions to clause or phrase boundaries with rules? Rule sieves fix parser debris in coreference (Illinois-Coref). But hallucination edges follow meaning, not syntax: an unsupported modifier inside a true clause is still the hallucinated part. We run a parse-based snapper as an additional corrector arm. If it matches the learned corrector, the cheaper tool wins and we will report it as the recommendation.

Why not just trust the synthetic insertions? This is the most serious threat. Inserted edits may leave stylistic seams that a detector could notice without consulting the source. We test for this with a no-source control in the spirit of Elazar et al.'s no-candidate baseline: the detector sees only the response. If its localization F1 on inserted items clearly exceeds its no-source F1 on natural RAGTruth items, the insertions are leaky. In that case we regenerate them with paraphrase smoothing until the no-source scores match.

Decisive Experiments

The study rests on three measurements.

First, we place the gaps among RAGTruth's top systems against the human-versus-human span F1 from the audit. If 55.9 and 51.2 both fall within the band of annotator disagreement, the current ranking is not statistically meaningful at the span level. That would be the Re-TACRED compression phenomenon, observed for the first time in hallucination detection.

Second, we apply the boundary-only corrector to o3 and to the SFT model and measure the fraction of their gap to RL4HS that disappears. We repeat this separately for each hallucination category, expecting the largest effect on subtle baseless information.

Third, we re-rank all systems on the fresh insertion benchmark using localization F1, and check two things: whether RL4HS's lead persists, and whether trained detectors drop more than prompted ones when moving from RAGTruth to fresh text. A larger drop for trained detectors would indicate reliance on RAGTruth-specific conventions or familiarity with its sources.

Any of the three outcomes is publishable. The gaps may prove to be real detection gains, they may be boundary convention, or the human ceiling may show that the benchmark cannot separate the systems. Each outcome directly changes how span-level hallucination detectors should be trained and compared.