Scoring Answers by How Well They Explain the Context: A Test of Whether Lost-in-the-Middle Is a Readout Artifact

Long-context language models answer badly when the fact they need sits in the middle of a long input. Accuracy is high when the fact is near the start or the end and sags in between, a U-shaped curve. Every published measurement of this effect asks the model to generate the answer after reading the whole input. There is evidence that the model often "knows" the answer anyway: the correct token is usually among its top eight candidates but loses to tokens copied from nearby text at the final step. We propose to read the answer out the other way around. We take a handful of candidate answers from the model itself, and for each one we ask how much more probable the context becomes when the model is told that candidate is true. This is the generative-classifier recipe, picking the label $y$ that best explains the input $x$, which has so far been used for images and short texts. Here it is applied to long-context retrieval and multi-hop question answering.

The idea fits in two sentences. If lost-in-the-middle is mainly a failure of the last generation step, then a likelihood-based readout should show a different position curve. With a prompt that restates the hypothesis before every document, that curve should be flat. Because this readout depends on how surprising the context is, it is exposed to a confound that generation is not: passages the model has memorized during pretraining carry almost no signal. So contamination has to be measured as part of the method, not left as an afterthought.

Core Claim

Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding used probes of intermediate layers to show that the gold token usually stays in the top eight predictions while nearby tokens win at decoding. It fixes this inside the decoder by subtracting a "near-sighted" logit baseline. Lost in the Middle: How Language Models Use Long Contexts established the U-curve, and Lost in the Middle, and In-Between showed a further drop when distractors are placed between two needed pieces of evidence. Both papers motivate our proposal. Neither changes the direction of inference.

We claim that the position curve depends on the readout. Suppose each candidate answer is scored by the evidence it adds to every context token, and the candidate is restated right before each document. Then the conditioning distance between hypothesis and evidence is constant, and the U-curve should mostly vanish, including the "in-between" penalty, which generation-based fixes leave in place. We also make a sharper prediction for the naive version, where the hypothesis appears once at the start. There the curve should decay monotonically with document position, with primacy but no recency. Seeing that shape would itself be evidence that position effects follow conditioning distance and not some fixed property of the middle.

Research Program

The method has two stages. First, sample or beam-decode $K=8$ candidate answers from the model's ordinary generation; the top-8 finding from the PCD paper suggests the gold answer is usually among them. Then rescore each candidate. Write $h_y$ for a short hypothesis sentence ("Assume the answer is $y$") and $h_\emptyset$ for a neutral sentence of matched length. With question $q$ and documents $d_1,\dots,d_n$, the score is

$$\Delta(y) = \sum_{i=1}^{n} \Big[\log p(d_i \mid q, h_y, d_{<i}) - \log p(d_i \mid q, h_\emptyset, d_{<i})\Big],$$

where, in the interleaved prompt, $h_y$ (or $h_\emptyset$) is inserted immediately before each $d_i$. The prediction is the candidate with the highest $\Delta(y)$. Subtracting the neutral term removes how easy each passage is to predict in general, much as diffusion classifiers reuse the same noise across classes (Your Diffusion Model is Secretly a Zero-Shot Classifier). We borrow that variance-reduction idea. We also borrow the "class token as prefix" construction for autoregressive text classifiers from Generative Classifiers Avoid Shortcut Solutions.

Restating the hypothesis before each document is the prompting contribution. It resembles divide-and-conquer prompting (When Splitting Makes Stronger), which shortens the context each sub-task sees. The difference is that we keep the full joint context, so that evidence spread across documents can still combine.

For synthetic key-value retrieval and RULER variable tracking, the candidate set is simply every value or variable name in the context, which makes this a closed-set generative classifier. Cost is $K+1$ long forward passes per question with no decoding. Interleaving breaks prefix caching, and we will report wall-clock time honestly. The diffusion-classifier literature documents up to 1,000 seconds per image on ImageNet, so the cost/accuracy trade-off is part of the result.

The contamination analysis is built into the design. Quantifying the Effect of Test Set Contamination on Generative Evaluations showed that a model trained on even one copy of a test set reaches lower loss than any clean model can, and that heavy contamination produces "deterministic lock-in." For a likelihood-gain score this is a direct threat. If a Wikipedia passage is memorized, $\log p(d_i \mid \cdot)$ is near its ceiling under any hypothesis, and $\Delta$ collapses toward zero. The standard multi-document QA set draws on Wikipedia, which is almost certainly in pretraining. We therefore treat memorization, measured as each passage's loss without the question, as a covariate, and we compare against counterfactual and synthetic contexts.

Baselines

The do-nothing reference is ordinary greedy generation from the same model and prompt, measured at every gold position. For Llama-3-8B, the PCD paper reports 64.40 F1 on RULER variable tracking at 16k tokens. For Llama-3-8B-262k it reports 72.0% on its InfiniteBench 8k task. We run the same models, lengths, and scoring scripts. For multi-document QA, Lost in the Middle reports GPT-3.5-Turbo at roughly 75% with the gold document first and roughly 53% in the middle. Our method needs token log-probabilities, so we cannot run it on that model. We cite those numbers only to describe the shape of the curve and regenerate the curve on open models.

The strongest inference-time competitor is Positional Contrastive Decoding. It targets the same diagnosis we do, decoding selection rather than representation, and it reports 69.04 F1 on RULER variable tracking at 16k (Llama-3-8B) and 79.0% on InfiniteBench 8k (Llama-3-8B-262k). We will rerun PCD ourselves on the same examples and tune its $\beta$ on the same development split and budget we use for our one free choice, the wording of the hypothesis sentence. Beating PCD means exceeding its rerun score on variable tracking and InfiniteBench, and also having a smaller gap between the best and worst gold position, at matched model and context length.

The strongest pipeline competitor is extractive reranking from Characterizing Prompt Compression Methods for Long Context Inference, which reports a 7.89-point gain on LongBench 2WikiMultihopQA at 7.75x compression. We run that reranker and the downstream generator on the same LongBench splits.

Two ablations isolate the key ingredient. The first, direct candidate scoring, reranks the same $K$ candidates by $\log p(y \mid x, q)$. It shows whether the gain comes from inverting the conditioning or merely from having a short list of candidates. The second, single-prefix channel scoring, places the hypothesis once at the start. It tests whether interleaving, and not channel scoring alone, is what flattens the curve.

Why not just…?

Why not just rerank the top-8 candidates by their direct likelihood given the context? This is the most serious threat, because the candidate list already does much of the work. If direct rescoring matches $\Delta(y)$, then the inverted readout adds nothing. Our position-curve plots for both methods, on the same candidates, settle it. If direct scoring inherits the U-shape and the interleaved channel score does not, the conditioning direction is the cause. If both are flat, we will say the benefit came from the candidate list.

Why not just compress the context with an extractive reranker? Reranking is strong on 2WikiMultihopQA. The same paper, however, warns that it is dangerous for tightly structured inputs such as SQL schemas, and the PCD paper notes that segment reranking breaks document flow. A reranker scores chunks one at a time, which works against "in-between" questions where neither chunk looks relevant by itself. We test this directly on the distance-controlled HotpotQA setup. If reranking matches us there too, our method's advantage is limited to settings where the context cannot be cut.

Why not just apply divide-and-conquer and score each document independently? For single-hop retrieval this would remove position effects completely, and it is cheaper to cache. We include it. For multi-hop questions, though, independent scoring cannot combine a bridge fact in one document with the answer in another. The divide-and-conquer paper itself states that the method fails when sub-problems are coupled. We expect a split result: per-document scoring ties our method on single-hop questions and loses on two-hop questions at distance greater than zero.

Why not just use chain-of-thought? In Lost in the Middle, and In-Between, chain-of-thought helped GPT-3.5 but hurt Llama-2-LongLoRA and did not remove the distance penalty. We include it as a baseline, but the file gives little reason to expect it to flatten the curve.

Decisive Experiments

The first experiment is the position sweep. We take synthetic key-value retrieval and 20-document QA with the gold item at each of 20 positions, on Llama-3-8B and a Qwen2.5-7B model. We plot accuracy against position for generation, direct rescoring, single-prefix channel scoring, interleaved channel scoring, and PCD. The predicted signature is a flat interleaved curve and a monotonically decaying single-prefix curve. Seeing a U-curve under interleaving would falsify the readout hypothesis.

The second experiment is the in-between test. On HotpotQA, 2WikiMultihopQA, and MuSiQue with controlled distance between the two evidence documents, the cited paper reports GPT-3.5 at about 50% at distance zero and 45% when separated. We measure the accuracy drop from distance zero to maximum distance for each method on our open models.

The third experiment tests contamination. We stratify multi-document QA by how memorized each gold passage is. We then replace passages with entity-substituted counterfactual versions in the style of NQ-Swap, and with fresh synthetic facts generated by DyVal's graph generator and embedded among distractors. Our prediction is that $\Delta(y)$ shrinks and accuracy falls on highly memorized passages, and that this disadvantage disappears on counterfactual and synthetic contexts.

For the controlled version, we continue pretraining a 1B-scale model on 0, 1, 10, and 100 copies of the evaluation passages, following the injection protocol of the contamination paper. We then measure how the channel signal degrades compared with generation, which that paper found can look better under contamination. If the channel readout loses accuracy under contamination while generation gains, that would make the readout useful as a contamination probe in its own right.

We report all results with bootstrap confidence intervals over questions, together with wall-clock cost per question.