Does Memorization Make Less Context Look Better? Context Budgets for Non-Factoid QA on Fresh Documents

Several recent results say that when a language model answers open-ended "why" and "how" questions over long documents, it does better with less of the document. It does better when a selector keeps only the few most relevant paragraphs, and it does better with a trimmed conversation history. We suspect part of this is an artifact. Most of these benchmarks are built from Wikipedia and similar web text, which the models have probably already memorized. A model that already knows the answer gains little from reading more, and extra text only distracts it. On a document the model has never seen, it has to assemble the explanation from the page itself. In that case the best amount of context may be much larger, and step-by-step reasoning may matter much more.

We propose to test this directly. We will build matched sets of non-factoid questions in which the only thing that changes is whether the model could have memorized the answer. On each set we will measure how answer quality varies with the amount of context provided, with and without step-by-step reasoning. If the effect is real, the practical guidance "prune aggressively and answer directly" rests on a biased evaluation. It would then need to be replaced by a context budget that depends on how much the model already knows.

Core Claim

We use four terms throughout. A non-factoid question asks for an explanation, a procedure, or a comparison rather than a name or number. Chain-of-thought (CoT) means the model writes out intermediate reasoning before answering. Benchmark contamination means test material appeared in pretraining data. The context budget $k$ is the number of top-ranked paragraphs given to the model.

Our claim has two parts. First, the context budget $k^*$ that maximizes answer quality is larger on uncontaminated documents than on contaminated ones. Second, the gain from CoT at a given $k$ is small on contaminated items and grows on uncontaminated ones, most of all at larger $k$, where the answer requires combining several paragraphs. We define the CoT gain $\Delta_{\text{CoT}}(k)$ as the quality with reasoning minus the quality with direct answering at the same budget.

This matters because the strongest evidence for aggressive pruning comes from a setting where contamination was never measured. Mishra et al. (Long-context Non-factoid Question Answering in Indic Languages) showed that plain top-5 paragraph selection beats full-document input by 47% in ROUGE and 4% in semantic similarity, and beats more elaborate extraction pipelines. They answered directly and did not check whether MuNfQuAD or the translated NaturalQA were in the training mixtures of Gemma and LLaMA. CORAL (CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation) found that citation quality degrades after 4–6 conversation turns, and its authors acknowledge that the Wikipedia source probably contaminates the benchmark. These two papers motivate our question. We do not dispute that their curves were measured correctly. We ask whether the curves transfer to text the model has not seen.

Research Program

The central artifact is a paired freshness testbed. Each question appears in three versions that share the same question and nearly the same document.

The original version uses the document as it appears in an existing non-factoid set: Wiki-NFQA from Typed-RAG, MuNfQuAD and the Indic NaturalQA from Mishra et al., and CORAL conversations.

The counterfactual version edits the document so that the correct explanation contradicts what the model would recall. Examples include swapping the stated cause of an event, reversing a step order in a procedure, or flipping which side of a comparison wins. The edits propagate consistently across paragraphs. At least one edited fact must sit in a paragraph that the relevance ranker places outside its top two, so that a good answer genuinely has to combine paragraphs.

The fresh version draws questions of the same type from news published after the models' training cutoffs, following the construction of CausalProbe-2024 (Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?). That paper built a post-cutoff causal set from BBC and Guardian articles. It found that models scoring near-perfectly on older causal benchmarks dropped sharply, and that standard CoT defaulted to memorized narrative clichés. We extend its approach from short causal pairs to multi-paragraph explanations over long documents.

The counterfactual version is what makes the design informative, because non-factoid answers are otherwise hard to score. Once an edit is planted, we can check objectively whether the explanation uses the document's version or the memorized version. We call the first rate edit adherence and the second memory leak. Neither depends on a judge model's taste.

For holistic quality we use MinosEval (MinosEval). It ranks each answer against graded reference answers and was designed specifically for non-factoid evaluation. We regenerate its reference answers from the edited document so that the judge does not reward memorized content. ROUGE is reported only for comparability with Mishra et al.

On every version we sweep $k$ from 1 paragraph to the full document, using the same paragraph selector as Mishra et al.'s best method. At each $k$ we run the model in three ways: answering directly, reasoning step by step, and reasoning step by step with an instruction to quote the paragraph behind each step. We also run a closed-book condition with no document. Its score, measured per item, is our direct estimate of how much the model already knows.

The analysis has three parts. We locate $k^*$ for each version. We plot $\Delta_{\text{CoT}}(k)$ across budgets. We regress both quantities on the closed-book score and on a membership-inference score computed with Min-K% Prob, the detector CausalProbe used.

A secondary deliverable follows only if the effect appears. It is a cheap controller that runs a closed-book probe first and uses the probe's score to choose $k$ and to decide whether to reason step by step.

Baselines

The naive reference points are closed-book answering and full-document direct answering at each model's maximum length. Closed-book answering sets the floor for what memorization alone achieves. It also defines the "do nothing" reference against which any context has to justify itself. Full-document direct answering is the condition Mishra et al. showed to be dominated on contaminated data. Both run on identical items and models: Llama-3.1-8B-Instruct, Gemma-7b-it, and Mistral-7B, the open models used by the cited papers. Each gets the same decoding settings and the same prompt-tuning budget, fixed at a small number of dev-set iterations per condition.

The strongest existing context-windowing method is top-5 answer paragraph selection with direct answering (A1 in Long-context Non-factoid Question Answering in Indic Languages). Its reported gain over full context is 47% in ROUGE and 4% in STS-MuTe on MuNfQuAD. We rerun it ourselves on all three versions rather than copying numbers, because those gains are relative figures on the original data only.

The strongest existing non-factoid method is Typed-RAG (Typed-RAG). It classifies the question type, splits the question into single-aspect sub-queries, retrieves for each, and aggregates the sub-answers. It reports an MRR of 0.7444 on the SQuAD-NF subset of Wiki-NFQA with Mistral-7B and a GPT-4o-mini judge, against 0.4222 for the model alone and 0.3817 for standard retrieval augmentation. It is the right competitor because it is effectively a structured reasoning procedure over pruned context. If it closes the fresh-versus-original gap by itself, our story is weaker. We run it with its published type set and the same retriever and paragraph pool as our sweep.

Two ablations isolate the key ingredient. The first compares original with counterfactual versions and holds everything else fixed, which separates memorization from domain and difficulty. The second replaces CoT with a matched-length filler, generic restatement tokens of equal count. This checks whether any CoT gain comes from reasoning content or merely from extra computation per token.

"Beating" a baseline means different things in different places. For the prescriptive controller, it means higher edit adherence and MinosEval rank than A1 on the counterfactual and fresh versions, at equal or lower total tokens, with the gain exceeding bootstrap confidence intervals. For the scientific claim, the claim is supported if $k^*$ on counterfactual items exceeds $k^*$ on original items for at least two of the three models, and $\Delta_{\text{CoT}}$ at large $k$ is significantly larger on counterfactual and fresh items.

Why not just…?

Why not just stratify existing benchmarks by a membership-inference score? This is the cheapest alternative and a real threat, so we run it. Min-K% Prob gives a per-item estimate of prior exposure, and CausalProbe used it to show that its fresh set was less exposed. But the score detects surface-level exposure of the document text, and it is correlated with topic popularity and difficulty. A correlation between that score and $k^*$ could therefore reflect easier questions rather than memorized answers. The counterfactual edit changes the answer while leaving topic, length, and difficulty nearly fixed, so it provides the causal contrast. If the stratified analysis reproduces the counterfactual effect with the same sign and similar size, we will say so and recommend the cheaper tool.

Why not just evaluate on post-cutoff news, as CausalProbe-2024 did? Fresh news differs from Wikipedia in style, structure, and question difficulty all at once. A shift in $k^*$ on news alone could reflect any of these. We keep the fresh version because it avoids the artificiality of edits, and we use the original-versus-counterfactual pair to attribute the effect. Agreement between the two uncontaminated versions is itself one of our checks.

Why not just give a long-context model the whole document and let step-by-step reasoning sort it out? Lin et al. (Facilitating Long Context Understanding via Supervised Chain-of-Thought Reasoning) argue that long-context comprehension is a reasoning bottleneck, not a capacity one. They report 69.58 on Loong for GPT-4o-mini with their agentic procedure, a 20.3% improvement. That supports reasoning over full context. However, their tasks are financial lookups on static benchmarks with unmeasured contamination. Our sweep includes full-document CoT as its right endpoint. If that endpoint dominates every smaller $k$ on fresh data, the practical conclusion becomes "stop pruning and reason," which still overturns the contaminated-data advice.

Why not just fine-tune the model to rely on context, instead of choosing budgets? Fine-tuning on documents that disagree with memory is a plausible fix. It is also a separate intervention that could hide the phenomenon we want to measure. We treat it as follow-up work, conditional on our effect appearing.

Proposed Experiments

The first decisive check is the $k$-sweep on the original and counterfactual versions of Wiki-NFQA and MuNfQuAD with direct answering. Our hypothesis predicts that original items peak near Mishra et al.'s five paragraphs and that counterfactual items peak later, with edit adherence rising in $k$ until the planted cross-paragraph edit is covered. If the curves coincide, the claim is false and the published pruning advice stands.

The second check adds step-by-step reasoning. The prediction is that $\Delta_{\text{CoT}}$ is near zero on original items, consistent with the "causal parrot" pattern CausalProbe reported. On counterfactual items it should be positive and growing in $k$, with the matched-length filler recovering little of it.

The third check repeats the sweep on the fresh news version and on CORAL-style multi-turn histories, replacing paragraph count with turn count. This tests whether CORAL's 4–6-turn degradation point moves later when the conversation's source text is edited.

The fourth check tests whether the closed-book probe predicts per-item $k^*$ well enough that the controller beats A1 and Typed-RAG on edit adherence and MinosEval rank at matched token cost. We report MinosEval's agreement with a small human-rated subsample, since its published Spearman correlation of 0.5489 on AlignBench_Minos was not measured on edited documents.