Language models now routinely summarize documents of tens of thousands of words. Checking those summaries for factual accuracy is hard. Word-overlap scores against a human reference miss most errors, and a single "judge" model asked to read the whole document gets worse as the document grows. The practical workaround is to cut the source into pieces and have a judge model check each claim in the summary against each piece. This note argues that the workaround brings back the very bias it was meant to remove. A judge that sees one piece at a time cannot tell "this fact is not in my piece" from "this fact is false." Claims that combine information from distant parts of the document, which is exactly what a good abstractive summary does, then get marked unsupported. The result is a hidden reward for copying local sentences and a penalty for synthesis. In effect, the segmentation becomes a narrow reference.
We propose to measure this "boundary-induced penalty" directly and to remove it with a simple change to how per-piece verdicts are produced and combined. Each window of source text returns one of three verdicts: supported, contradicted, or absent here. It also points to the source spans that partially support the claim. When no single window fully supports a claim but several support parts of it, the cited spans are stitched together in document order and judged once more. Few-shot demonstrations (in-context learning, where worked examples are placed in the prompt instead of training the model) teach the judge the three-way distinction, which judges do not draw reliably when only told about it. The contribution is a diagnostic and a fix. The diagnostic is a measurable form of reference bias that arises from segmentation rather than from a written reference. The fix is a verdict-and-aggregation protocol whose benefit can be isolated by ablation.
Core Claim
Segmented semantic evaluation of long documents has a systematic locality bias. The probability that a true claim is marked unsupported rises with the distance between the source spans that jointly support it. The bias is largest for summaries that synthesize, so it distorts system rankings in favor of extractive outputs. Two changes together should flatten the false-penalty curve without the cost and degradation of feeding the whole document to one judge. The first is separating absence from contradiction at the window level. The second is a targeted re-judging step over stitched evidence.
How This Relates to Prior Work
The motivation comes from the reference-bias literature. Single references deflate scores of valid alternative outputs. Exact match against gold answers understates open-QA accuracy by nearly ten points (59.2 vs 68.9 human, DPR+FiD on Natural Questions) in Evaluating Open-QA Evaluation. Showing a reference to a rater makes them penalize valid divergent phrasing, according to A Critical Look at Meta-evaluating Summarisation Evaluation Metrics.
Three short-document methods attack reference bias by changing what the candidate is compared against. QAPyramid decomposes references into question-answer pairs and awards partial credit. HighRES replaces references with human highlights on the source. RevisEval rewrites the candidate into a response-adapted reference for a judge. All three operate on texts of roughly 500 words, where the whole source fits in one view, so none can exhibit the failure studied here. We borrow QAPyramid's move from whole-text to atomic-unit comparison, applied to the candidate's own claims, and HighRES's source-grounding. We do not build any written reference.
The segmentation methods supply our window construction, and their limits motivate the second stage. These are the 50% sliding-window overlap in InstructCoder and persistent headers prepended to every chunk in Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations. Both bridge dependencies only up to the overlap width. A claim whose support lies 20 pages apart is invisible to any single overlapping window.
The long-context literature constrains the design. LongBench reports a 17% accuracy drop for GPT-3.5-Turbo-16k from 0–4k to 8k+ inputs, and L-Eval reports instruction-following failures beyond 4k tokens. Together they argue against the monolithic judge, though not decisively; that has to be tested. L-Eval's fix for length bias forces candidates to match reference length. That repairs a correlation statistic and leaves factual blindness in place, which FRANK puts at a maximum Pearson correlation of about 0.20 between factuality metrics and human judgment.
Finally, Favi-Score supplies the statistical lens. A metric can have good per-sample accuracy while systematically favoring one class of systems. Our hypothesis is that segmented judges favor extractive systems, and we use Favi-Score to quantify that.
Research Program
The pipeline has three stages, each deliberately simple.
The first stage breaks the candidate into atomic claims with an LLM prompt, in the spirit of QAPyramid's units but drawn from the candidate. This makes it source-grounded and reference-free for factuality.
The second stage builds windows with 50% overlap and a persistent document header. A judge then gives each window, as a batch, all claims plus few-shot demonstrations. For each claim, the judge returns supported, contradicted, or absent here, along with the span offsets of any supporting fragment and a flag saying whether that fragment covers the whole claim. The demonstrations deliberately include near-miss cases. One kind is a claim whose entity appears in the window but whose attribute does not, which should get absent rather than contradicted. Another is a claim half-supported in the window, which should get partial.
The third stage aggregates. A claim with any full support and no contradiction is supported. A claim with any contradiction, confirmed by a re-judgment on the contradicting span plus header, is contradicted. A claim that is absent everywhere is unsupported. A claim with only partial supports goes to stitching: the cited spans, each with a small surrounding margin and the header, are concatenated in document order and judged once. Cost grows linearly in windows, plus a stitch call for the small fraction of partially supported claims.
The analysis side is as important as the method. We define the boundary-induced penalty as the false-unsupported rate on claims that human annotators mark true. We plot it against the token distance between their supporting spans. We also define a locality-favoritism score using Favi-Score's directed-error formulation, contrasting systems grouped by the fraction of their claims that require multi-span support. In-context learning plays the role of the cheap controller for the abstention behavior. The ablations ask whether demonstrations are necessary or whether an instruction alone gets the judge to separate absence from falsity.
Baselines
The do-nothing reference point is disjoint chunking with binary per-chunk verdicts. A claim counts as supported if any chunk says so, and otherwise as unsupported. This is the common practice, the equivalent of MuLD's chunked T5 setup (MuLD reports 54.01 vs 82.58 F1 for chunked versus sliding-window models on character classification, a reminder of how large boundary costs can be). Beating it is the minimum bar.
The stronger segmentation baseline is the same binary judge run over 50% overlapping windows with persistent headers, following InstructCoder and Zhang et al. This isolates whether overlap alone removes the penalty. If our method only beats it at distances greater than the overlap width, that confirms the mechanism.
The strongest existing competitor is the monolithic long-context judge. One frontier model with a 128k–200k context receives the full document, the candidate, and the same claim list and demonstrations. It is strongest because it never cuts the document, and at system level L-Eval reports that a GPT-4 judge reaches a Kendall-Tau of 1.0 with human rankings under length-instruction-enhanced evaluation (L-Eval). Because that number is saturated at system level, our comparison is at claim level, where no reported result found on Lacuna for claim-level factual verification over 10k+ token documents. The nearest short-document anchor is GPT-4 at 82.4% balanced accuracy on SummEdits, against 90.9% for humans.
The per-claim retrieval baseline embeds the claim, retrieves the top-k chunks, and judges once. It gets the same judge model and a k chosen so its token budget matches ours.
The length-constrained overlap baseline is ROUGE-L under L-Eval's length instruction, the cheapest serious alternative.
Fairness rules apply to every LLM-based condition. All use the same judge model, the same claim list, the same demonstrations where applicable, and a matched prompt-tuning budget (the same number of prompt variants tried on a held-out development split). We report total tokens processed per candidate.
Beating the field means two things. On the controlled test, our false-unsupported rate on true claims must stay flat across distances where the overlapping baseline rises. On natural documents, claim-level balanced accuracy must match or exceed the monolithic judge at lower token cost, with a lower locality-favoritism score.
Two ablations isolate the key ingredient. The first removes the absent label, so verdicts become binary but stitching is kept. The second keeps the three-way verdict but removes stitching. A third ablation replaces demonstrations with instruction-only prompting, to test whether in-context learning is doing real work.
Why not just…?
Why not just give the whole document to a long-context judge? This is the real threat, and we treat it as one. LongBench and L-Eval document degradation with length, but those results concern task performance, not claim verification by the strongest current models, which may have improved. The controlled distance experiment settles it. If the monolithic judge's false-penalty curve is as flat as ours at all document lengths we test, the method's case shrinks to cost. We would then report it as a cheaper equivalent rather than a more accurate one. Our prediction is that the monolithic judge degrades with total document length even when support distance is fixed. That is a separate failure from the segmented judges, which degrade with support distance even when length is fixed.
Why not just retrieve the relevant chunks per claim? Retrieval fixes the distance problem only when every supporting span resembles the claim on the surface. Cross-segment support often arrives through coreference ("the firm" thirty pages after the company is named) or through an attribute stated without the entity. LongBench reports that retrieval hurts models with native long-context ability by dropping distributed information. Our stitching step retrieves by the judge's own partial-support pointers rather than by embedding similarity. The controlled test includes a coreference condition, where one supporting span never names the claim's entity, to separate the two.
Why not just use larger overlaps? Overlap bridges only distances shorter than the overlap. Growing it toward document length turns the method into the monolithic judge at several times the cost. The distance curve shows exactly where each overlap setting breaks.
Why not compress the demonstrations into a task vector, as in In-Context Learning Creates Task Vectors? That paper shows that hidden states from demonstrations, injected into a query-only pass, retain 80–90% of in-context accuracy on single-token tasks. Our verdict is nearly single-token, so the idea is tempting. However, a 10–20% loss would swamp the effect we are measuring, and reusing a cached demonstration prefix gives the same compute savings with no accuracy loss. We therefore leave task vectors out of the method and mention them only as a possible later cost study.
Proposed Experiments
The decisive experiment is a controlled distance test built from SummEdits. Its 6,348 human-labeled consistent and inconsistent summary-document pairs span ten domains, including QMSum meetings, BillSum legislation, and sales calls. Each source document is embedded in a 20k–60k token host built from other same-domain documents. We take claims whose support involves two sentences, and move one of the two supporting sentences to a controlled distance from the other, with optional pronoun substitution for the coreference condition. We then measure the false-unsupported rate on true claims and the miss rate on edited false claims as functions of distance and total length.
Because the labels come from SummEdits (kappa 0.9), the ground truth is trustworthy. The manipulation only relocates text, so it should not change the truth value; a small human audit checks that it does not.
The natural-data experiment uses long-document summarization from MuLD's very long scientific papers and L-Eval's open-ended tasks. Outputs come from several systems, chosen to span extractive and abstractive styles. Annotators label atomic claims as supported or unsupported with the full source available, along with how many source locations each claim draws on. A modest set of around 1,500 claims is feasible at QAPyramid-scale effort. We report claim-level balanced accuracy, the boundary-penalty curve on naturally occurring multi-span claims, the Favi-Score locality favoritism, and system-level Kendall-Tau against human rankings. System-level correlation is included because a bias that does not change rankings matters less, and showing that rankings shift is the practical payoff.
The result that would falsify the core claim is a flat false-penalty curve for the overlapping-window baseline on natural data. That would mean multi-span claims are too rare, or too close together, for locality bias to matter in practice. The result that would make the method unnecessary is a monolithic judge that matches it at every length.
References
- Evaluating Open-QA Evaluation
- A Critical Look at Meta-evaluating Summarisation Evaluation Metrics
- QAPyramid: Fine-grained Evaluation of Content Selection for Text Summarization
- HighRES: Highlight-based Reference-less Evaluation of Summarization
- RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
- InstructCoder: Instruction Tuning Large Language Models for Code Editing
- Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- L-Eval: Instituting Standardized Evaluation for Long Context Language Models
- Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics
- Favi-Score: A Measure for Favoritism in Automated Preference Ratings for Generative AI Evaluation
- MuLD: The Multitask Long Document Benchmark
- SummEdits: Measuring LLM Ability at Factual Reasoning Through The Lens of Summarization
- In-Context Learning Creates Task Vectors