Long-context benchmarks compare language models whose input limits range from a few thousand to over a hundred thousand tokens. When a document is too long for a model, the evaluation code cuts it down, usually by keeping the beginning and end and deleting the middle. This is called input truncation. Each model is therefore scored on a different amount of text, and the dropped part may or may not contain the answer. Nobody records which. We propose to measure, instance by instance, whether the evidence a question needs survives truncation. We then use that record to split each model's score into two parts: failures caused by the harness deleting the evidence, and failures of the model itself.
The motivation comes from other fields. When evaluation pipelines were standardized there, reported advantages repeatedly turned out to be artifacts of engineering choices. In recommender systems, tuning old baselines as hard as new ones erased most deep-model gains (BARS). In forecasting, dropping the last incomplete test batch lowered reported error (TFB). In view synthesis, image-resizing details flipped rankings (NerfBaselines). Truncation policy is the equivalent uncontrolled variable in long-context NLP, and it is currently hidden inside benchmark code.
Core Claim
The claim is that part of what long-context leaderboards report as differences in model ability is really evidence survival: whether the facts needed for the answer remained after the harness cut the input. We will show this by annotating where the evidence sits in each document. With those annotations we can compute, for any model, policy and length limit, the probability that the evidence survives. The score then splits as
$$\text{Acc} = s \cdot A_{\text{kept}} + (1-s)\cdot A_{\text{lost}},$$
where $s$ is the fraction of instances whose evidence survives truncation, $A_{\text{kept}}$ is accuracy on those instances, and $A_{\text{lost}}$ is accuracy on instances whose evidence was deleted. $A_{\text{kept}}$ is the quantity a benchmark claims to measure. $A_{\text{lost}}$ should roughly equal the model's accuracy with no context at all; any excess signals leakage or answers the model already knows from pretraining. Two models can post the same headline score with very different $A_{\text{kept}}$, and a ranking can flip when only $s$ changes.
Relation to Prior Work
Two lines of work motivate the proposal.
The first is benchmarks that fix truncation without auditing it. LongBench builds 21 tasks in a unified format. It applies middle truncation by default and reports length buckets (0–4k, 4k–8k, 8k+), including a 17% accuracy drop for GPT-3.5-Turbo-16k from the shortest to the longest bucket. That drop mixes lost evidence with genuine degradation over length, and our decomposition separates the two. HELMET evaluates 51 models up to 128k tokens with a model-based judge. It feeds contexts whole, so models with short windows need external preprocessing that HELMET does not specify; we specify and audit it.
The second line supplies partial methods. 100-LongBench builds length-controllable inputs, filters out questions answerable from pretrained knowledge, and introduces LongScore, which separates short-context ability from change with length. It shows that rankings reshuffle under this metric. It is the closest existing work. However, it evaluates only models that natively accept the full input. It never asks what happens when the harness deletes text, and it does not condition on evidence location. We reuse its no-context filtering idea as the reference point for $A_{\text{lost}}$.
A Controlled Study on Long Context Extension fixes the base model, the data and the hyperparameters across context-extension methods, producing a family of same-base models with different effective windows. We borrow this discipline and, where checkpoints are available, the model family. That study holds truncation implicitly constant rather than varying it. LongGenBench constrains our scope. It shows that a model's own accumulated output degrades accuracy by up to 47.1% for LLaMA-3-8B, so we restrict ourselves to short-answer tasks where output length cannot confound the input-side measurement.
On the encoder side, the file records a tension between two findings. Cohan et al. find flat joint attention across sentences beats hierarchical encoders on abstracts that fit in the window. Lu et al.'s hierarchical BERT and Ji et al. find that chunked or hierarchical models win on longer inputs. We read this as a possible truncation artifact: the flat baselines in the long-document papers were run with a single truncation policy. We test that reading in a secondary study.
Research Program
The first component is an evidence-location layer added on top of existing benchmarks.
For NIAH-style and RULER tasks, the needle position is known by construction.
For multi-document QA in LongBench, the gold document can be identified. We locate it by matching the answer string, confirm with a leave-one-document-out check (does removing the document change a strong model's answer?), and have humans verify a random sample.
For single-document QA, we locate answer-bearing sentences the same way. We mark instances with diffuse evidence as "distributed" and report them separately rather than forcing a location.
Summarization is excluded, because its evidence is inherently global.
The second component is a truncation harness. It applies each of five policies to every model at its native limit and at shared cutoffs of 4k, 8k and 16k tokens: head-only, tail-only, middle removal (the LongBench default), uniform sentence subsampling, and a query-aware sentence ranker. For each combination of policy, cutoff and instance it logs whether the evidence survived. The query-aware ranker is included only because it is the strongest "smart truncation" competitor, not because it is a contribution.
The third component is a causal intervention. For multi-document QA we permute document order so that the gold document lands in the region a given policy keeps or in the region it drops. Content and length stay fixed. This converts survival from an observed correlate into a controlled treatment and makes $A_{\text{kept}}$ and $A_{\text{lost}}$ directly estimable.
The model population deliberately spans window sizes: Llama-3.1-8B, Qwen 2.5-7B and 14B Instruct, and same-base context-extended variants (NTK-32k, YaRN) trained as in the controlled study. Where compute allows, we add Doc-to-LoRA on Gemma-2-2b as a system that avoids truncation altogether by converting the document into adapter weights. Doc-to-LoRA reports 82.5% of full-context performance on SQuAD. Our decomposition asks whether its shortfall comes from lost evidence or from misuse of evidence it retained.
The secondary encoder study reruns flat RoBERTa on the long Movie Reviews setting under all five policies and compares it with the hierarchical model. It asks whether the reported hierarchical advantage (0.9139 vs 0.8878 AUC at 50 labels) shrinks once the flat model gets its best truncation policy.
Baselines
The naive reference point is the status quo: each model evaluated under LongBench's fixed middle truncation at its native limit, with only headline accuracy reported. Every comparison uses the same instances and the same prompts. Its per-task scores are what we will recompute; the file has no reported per-subtask numbers ("no reported result found on Lacuna for LongBench subtask scores"), so we regenerate them ourselves. For this question, the only reported LongBench figure is the 17% bucket drop for GPT-3.5-Turbo-16k. Our method "beats" the status quo if the decomposition predicts results it cannot. Concretely, $s$ should explain a substantial share of cross-policy score changes, and an ordering by $A_{\text{kept}}$ should correctly predict model order under the permutation intervention where headline accuracy fails to.
The strongest existing method for isolating true long-context ability is LongScore from 100-LongBench. The file gives no reported numeric LongScore values ("no reported result found on Lacuna for absolute LongScores"). We will compute LongScore for every model on the same instances. The comparison asks which ranking better predicts held-out behavior on permuted instances, LongScore or $A_{\text{kept}}$. Beating LongScore means higher rank agreement with the intervention outcomes. If LongScore agrees equally well, we will say that it already captures the effect.
A second strong baseline is equal-length evaluation: every model truncated to the smallest shared window under a single policy. This removes differences in $s$ by construction. The ablation that isolates our key ingredient, evidence annotation, is to rerun the decomposition with random pseudo-locations in place of the true evidence spans. If conditional accuracies stay informative under random locations, the annotation is doing nothing.
For the encoder study, the baseline is the hierarchical model's reported 0.9139 AUC (Lu et al.). The flat model gets an identical tuning budget per policy, applied to all five policies.
Why not just…?
Why not just use a native long-context model and stop truncating? Long windows are not used reliably. LongBench's 17% drop and HELMET's reported degradation of open models at 128k show this. In addition, heterogeneous leaderboards will keep containing short-window models. Our native-window condition directly measures what full context buys each model, so the experiments settle this rather than assume it.
Why not just report LongScore? This is the real threat. LongScore already reshuffles rankings by separating short-context ability from change with length. It cannot, however, distinguish a model that degrades from a model whose harness deleted the answer, because it never observes evidence location. The head-to-head comparison against the permutation intervention decides whether that distinction matters in practice.
Why not just truncate every model to the same length? Equal-length evaluation fixes comparability but answers a narrower question, since it discards whatever a longer window genuinely contributes. It also still depends on which policy is chosen, and head versus tail at the same length can change $s$ sharply. We include it as a baseline and report whether rankings under it agree with $A_{\text{kept}}$ at native limits.
Why not just retrieve the top chunks? LongBench's own ablation reports that retrieval hurts models with strong native long context. Retrieval is also simply another truncation policy, one with a learned selector. Our query-aware ranker covers it, and the decomposition applies to it unchanged.
Proposed Experiments and Decisive Checks
The first check is ranking stability. We compute Kendall's tau between model rankings under each pair of policies, along with the fraction of model pairs whose order flips. We also check whether ranking by $A_{\text{kept}}$ is stable across policies where raw accuracy is not. If raw rankings are already stable (tau above 0.9, almost no flips), the confound hypothesis fails, and we will publish that as a clean negative result that licenses current practice.
The second check is explanatory power. We regress per-model score changes across policies on changes in $s$. A high share of variance explained means the harness, not the model, drives those differences.
The third check tests the parametric-knowledge reading. We compare $A_{\text{lost}}$ with no-context accuracy for each model. A gap indicates that answers leak from surviving text or pretrained memory, which would also challenge the QA filtering used by 100-LongBench.
The fourth check is causal. Under document permutation, $A_{\text{kept}}$ should predict accuracy when the evidence is placed in the kept region, and no-context accuracy should predict accuracy when it is placed in the dropped region.
The fifth check is the encoder study. If the flat model under its best policy closes most of the gap to the hierarchical model's 0.9139 AUC, the flat-versus-hierarchical tension in the literature is partly a truncation artifact. If the gap persists, hierarchy earns its reported advantage.
References
- BARS: Towards Open Benchmarking for Recommender Systems
- TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods
- NerfBaselines: Consistent and Reproducible Evaluation of Novel View Synthesis Methods
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly
- 100-LongBench: Are de facto Long-Context Benchmarks Literally Evaluating Long-Context Ability?
- A Controlled Study on Long Context Extension and Generalization in LLMs
- LongGenBench: Long-context Generation Benchmark
- Pretrained Language Models for Sequential Sentence Classification.
- A Sentence-Level Hierarchical BERT Model for Document Classification with Limited Labelled Data
- Applying Contrastive Learning to Code Vulnerability Type Classification
- Doc-to-LoRA: Learning to Instantly Internalize Contexts