Benchmarks for long documents increasingly hand their scoring to a large language model acting as a grader, an "LLM judge." The judge reads a model's long summary or answer alongside the source material and decides whether it is correct. We already know that language models reading long inputs neglect whatever sits in the middle. That defect has been measured in the models being graded but not in the models doing the grading. This proposal tests one specific risk. A judge reading a 5,000-word summary may catch a fabricated fact in the first or last paragraph and miss the same fabrication in paragraph twelve. If so, long-context leaderboards systematically reward any generator whose mistakes happen to fall where the judge is not looking.
We will build a contamination-free audit for this. We generate fictional source corpora with known facts, plant one controlled error at a chosen position in a candidate output, and record how often the judge catches it as a function of where the error sits and where its evidence sits in the source. We then measure how much this blind spot changes model rankings on existing long-context benchmarks. Finally, we compare cheap fixes head to head, including a small judge fine-tuned on a compact, position-balanced error set. The question is whether any of them closes the gap.
Core Claim
Our claim is that LLM judges for long outputs have a measurable positional blind spot, and that it compounds with a known generator failure. Lost-in-the-Middle in Long-Text Generation shows that generators leave a "middle void," under-covering centrally located source content. That paper's own consistency score relies on an LLM judge that it assumes is immune to the same effect. If judges also under-detect omissions whose evidence lies mid-source, the two failures hide each other. Benchmark scores would then overstate long-context ability exactly where it is weakest.
The quantity we care about is the positional detection gap:
$$G(L) = \tfrac{1}{2}\big[D(0, L) + D(1, L)\big] - D(0.5, L),$$
where $D(p, L)$ is the probability that the judge flags a planted error located at relative position $p \in [0,1]$ within an input of length $L$ tokens. We compute $G$ separately for the error's position in the candidate output and for the position of its supporting evidence in the source. A judge that is not biased by position has $G = 0$. The claim is falsified if $G$ stays near zero for strong judges up to 128K tokens.
Why This Gap Is Open
Three lines of work motivate the question without answering it.
The first studies position bias in judges, but at short range. Judging the Judges studies which of several short candidates a judge prefers and finds that bias peaks when candidates are close in quality. The Silent Judge shows that judges follow metadata cues such as recency and provenance and never admit to doing so. Both concern the order of candidates or labels attached to them, not where within one long document an error sits.
The second line studies position effects at long range, but only in the model being graded. Lost in the Middle shows multi-document QA accuracy for GPT-3.5-Turbo falling from about 75% to about 53% when evidence moves to the middle. Positional contrastive decoding traces this to nearby tokens suppressing distant correct ones. Neither paper examines a model in the grading role.
The third line deploys long-context judges without auditing them. HELMET replaces ROUGE with GPT-4o for long QA and summarization at up to 128K tokens. LONGINOUTBENCH uses an LLM judge to verify 1,200 fact pairs inside multi-thousand-word summaries. Our audit tests the assumption both of these benchmarks make.
Contamination enters our design as a constraint. If a judge has seen the source facts during pretraining, it can grade from memory and never consult the middle of the context, which would mask the effect we want to measure. Static long benchmarks also age into training corpora, which is why BAMBOO restricts itself to post-2023 text. We go further and generate the facts procedurally, in the spirit of DyVal. DyVal builds reasoning problems from randomly generated graphs but stays in short formal domains, and we extend the idea to long narrative documents.
Preference Leakage adds a second constraint. Judges score text from related models more generously, so the model that writes our fictional worlds must come from a different family than any judge under test.
Research Program
Audit construction. We sample a random fact graph of fictional entities: people, organizations, dates and quantities, with relations among them. From it we render a source corpus of 8K to 128K tokens and a faithful reference summary of 1K to 8K words. Rendering uses a generator from a model family excluded from the judge pool, followed by a paraphrase pass so that surface style carries no signal.
Each audit item is the reference with exactly one perturbation, of one of three types: a contradicted fact, a fabricated fact, or an omitted fact. We control two positions independently: where the perturbation sits in the candidate, and where the relevant evidence sits in the source. Clean items, which contain no perturbation, measure false alarms. Because fresh worlds can be generated at any time, the audit cannot be memorized. We will also release a generator rather than a fixed test set.
Realism check. Synthetic worlds could exaggerate or hide the effect. We therefore also inject the same three perturbation types at controlled positions into real generations on HELMET summarization and LONGINOUTBENCH. We hand-verify a subset of several hundred items and check that the $G$ estimates from synthetic and real items agree.
Ranking distortion. We score a pool of open and closed generators on LONGINOUTBENCH and HELMET summarization with each judge. For every generator, we then compare two quantities. The first is where its true errors lie, measured with the human-verified subset and per-fact checks. The second is how much the judge's score overstates that generator's quality. The prediction is that generators whose errors cluster mid-output gain the most, and that corrected scoring reorders the leaderboard.
Fixes. We compare four ways to correct the judge. The first is decomposition, which splits the candidate into atomic claims and checks each one separately. The second is rotation averaging over source documents. The third is positional contrastive decoding applied to the judge at inference. The fourth is a small trained judge.
The trained judge carries the data-quality question. We fine-tune ProLong-8B, a long-context model built from a small curated data mix that used 5% of Llama-3.1-8B's long-context token budget, on a few thousand error-injection items. These items are balanced over positions and concentrated on the hardest cases, which are near-miss contradictions placed mid-document. This mirrors Lion, which shows that a compact set targeted at a student model's failures beats volume. Training worlds and test worlds use disjoint fact templates and domains, so that a gain cannot come from learning the generator's habits.
Baselines
The naive reference point is lexical overlap, ROUGE-L against the reference summary. It has no positional attention, so it should show $G \approx 0$, but its overall detection should be poor. HELMET reports that ROUGE masks repetition and incoherence. Including it shows what a judge gains overall and what it loses in the middle. We consider it beaten when a judge has higher overall detection than ROUGE-L at matched false-alarm rate.
The strongest existing method is the GPT-4o judging protocol from HELMET, run with its published prompts. We also run the LONGINOUTBENCH consistency judge as published. Both are what current long-context leaderboards actually use, so any fix has to beat them. Every judge sees identical items, identical context lengths and the same false-alarm calibration on clean items.
We found no reported result on Lacuna for how well either judge detects errors as a function of position. The closest positional number in the file comes from short-form MTBench, where GPT-4 and Claude-3.5-Sonnet have a position consistency of around 0.82 (Judging the Judges). For downstream context, RAL-WRITER reaches a consistency score $S_c$ of 55.28 on Qwen2.5-14B (Lost-in-the-Middle in Long-Text Generation). Beating the strongest baseline means two things. First, $G$ must drop below the GPT-4o judge's value with confidence intervals that exclude it. Second, overall detection at matched false-alarm rate must be no worse.
Two ablations isolate the key ingredient of the trained judge. The first trains on the same number of error items with errors placed where natural generations put them, which over-represents boundaries. This separates position balancing from simply training on errors. The second trains on a ten-times-larger generic judge dataset with no positional control. This tests whether data volume can stand in for targeted coverage. All trained variants share the same base model, number of optimizer steps and tuning budget.
Why not just…?
Why not just decompose the output into atomic claims and verify each one? VALOR-EVAL uses this kind of two-stage parse-then-match judging, and it is the most serious threat to the trained judge. Decomposition removes the candidate-side position problem by construction. It leaves two weak points. The claim extractor itself reads the long output and may drop middle claims. Each claim check still searches a long source, so evidence-side position remains. Omissions are also invisible to claim-by-claim checks unless the source is decomposed as well, which is costly. Our audit measures $G$ for decomposition separately on the candidate side and the source side, and reports its cost. If decomposition closes both gaps at acceptable cost, we will say so, and the contribution reduces to the audit and the ranking correction. That would still be a useful result.
Why not just shuffle the source documents and average the verdicts? Averaging is the standard fix for pairwise position bias. It applies only to multi-document sources, because a single narrative cannot be shuffled without changing its meaning. Averaging also spreads a blind spot across positions rather than removing it. Rotation can help with evidence-side gaps when the error lands at a boundary in some permutations, and our experiment varying source position quantifies exactly how much. It costs a multiple of the judging budget and does nothing for errors located in the candidate.
Why not just use the strongest frontier judge, or humans? Frontier judges may already be immune, and that is a legitimate possible outcome. Only the audit can establish it. HELMET documents that open models trail closed ones by 30 to 40 points at 128K on citation and re-ranking, so judge choice plausibly matters and must be measured rather than assumed. Human grading of 8K-word summaries against 128K-token sources is slow and itself subject to fatigue. For that reason we use humans only to validate a subset.
Why not just use program-based judges? PAJAMA turns judging into deterministic Python scripts, which removes prompt-order bias. Its own authors note that the scripts struggle with open-ended semantic equivalence. Checking whether a paraphrased claim contradicts a fact buried 60K tokens into a source is the kind of task those scripts handle poorly, so we include PAJAMA only as a comparison on the synthetic split, where facts are structured.
Proposed Experiments
The first decisive check is whether $G > 0$ for the HELMET GPT-4o judge and for open judges. We measure it at 8K, 32K and 128K source lengths and at 1K and 8K candidate lengths, separately for each error type, with bootstrap confidence intervals over fresh worlds. We expect omission detection on the source side to show the largest gap, because omissions require the judge to scan the entire source.
The second check is synthetic-to-real agreement: $G$ estimated on fresh worlds should match $G$ estimated on the human-verified injected real generations to within its confidence interval.
The third check is ranking distortion. We compute the rank correlation between judge-based and position-corrected leaderboards on LONGINOUTBENCH and HELMET summarization. We also correlate each generator's mid-output error share with its score inflation. A positive correlation is what would make the blind spot matter in practice.
The fourth check compares all fixes against the baselines above on held-out world templates. We report $G$, overall detection at matched false-alarm rate, and cost per item. The ablations show whether a small position-balanced set beats both a natural-position set and a ten-times-larger generic set.
References
- Lost-in-the-Middle in Long-Text Generation: Synthetic Dataset, Evaluation Framework, and Mitigation
- Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge
- The Silent Judge: Unacknowledged Shortcut Bias in LLM-as-a-Judge
- Lost in the Middle: How Language Models Use Long Contexts
- Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding
- HELMET: How to Evaluate Long-context Models Effectively and Thoroughly
- BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Models
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Preference Leakage: A Contamination Problem in LLM-as-a-judge
- How to Train Long-Context Language Models (Effectively)
- Lion: Adversarial Distillation of Proprietary Large Language Models
- VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models
- Time to Impeach LLM-as-a-Judge: Programs are the Future of Evaluation