Counterfactual Document Edits to Measure the Real Open-Model Gap in Query-Focused Summarization

When a language model summarizes a meeting or a table in response to a user's question, we would like to know whether it read the document or recalled something it saw during training. On public benchmarks we cannot tell. The source texts and gold summaries are on the web, so a high score may reflect memory rather than reading. This matters most when comparing open-weights models with proprietary ones. That comparison is usually reported as a single score gap, but the gap could be inflated or hidden if the two model families have absorbed different amounts of the test data. We propose a simple, black-box test. Take an existing query-focused benchmark, make small controlled edits to the source document, and update the reference summary so it stays correct. Then check whether each model's summary follows the edited document or falls back to the original.

The core idea in two sentences: if you change a fact in the part of a meeting transcript that answers the user's question, a model that is actually reading should change its summary, and a model that is recalling should not. If you change a fact the question does not depend on, a model that is actually focusing on the question should leave its summary alone. Measuring these two behaviors separately, across open and proprietary models, gives a version of the open-model gap that is resistant to contamination. It also shows whether open models lag because they read worse, focus worse, or memorize more.

Core Claim

We claim that the open-versus-proprietary gap on query-focused summarization (QFS), meaning summarizing a document with respect to a specific user question rather than generically, is not one number. It splits into three measurable components. The first is grounding: does the summary follow edits to query-relevant evidence? The second is focus: does the summary stay unchanged under edits to irrelevant text? The third is recall leakage: does the summary revert to the original fact or reproduce the original gold wording?

We expect the gap to widen on edited documents relative to the original benchmark. INVITE showed this pattern in a much simpler setting: on false-premise questions, GPT-4 hallucinated 6% of the time while Open-LLaMA and GPT-J exceeded 80% (INVITE). Whether this pattern holds for long-form, query-conditioned synthesis is untested. The hypothesis fails if the edited-document gap equals the original gap within confidence intervals.

Relation to Prior Work

Three lines of work motivate the proposal.

First, contamination-free evaluation exists, but not for QFS. DyVal generates reasoning problems from random graphs and found that open models with strong static scores (phi-1.5, WizardMath-13B) scored nearly zero on its generated tasks. Its own authors note the method does not extend to unstructured long-form text. INVITE distorts knowledge-base triples into invalid single-hop questions. We borrow the logic of both, controlled perturbation with a known correct answer. We apply it to natural documents rather than synthetic graphs or single facts, because generating realistic meetings from scratch is exactly what DyVal could not do.

Second, the closest existing work joining summarization, contamination, and the open-model gap is FABLES. It avoids contamination by using 26 books published in 2023–2024 and spends $5.2K on 3,158 human claim annotations. It found Mixtral on par with GPT-3.5-Turbo and Claude-3-Opus most faithful. FABLES also constrains our design: the best automatic judge reached only 58.2% F1 at flagging unfaithful claims, so we cannot rely on open-ended LLM judging. Our protocol differs from FABLES in three ways. It is query-focused rather than generic. It reuses existing benchmarks rather than requiring fresh texts. Because we know exactly which fact was changed, checking an output reduces to a targeted lookup rather than open-ended fact verification.

Third, Data Contamination: From Memorization to Exploitation supplies the conceptual distinction we need. In controlled BERT experiments, about 15% memorization coexisted with under 1% exploitation, while duplicating a test example 200 times raised exploitation to as much as 40%. That work needed white-box control of pretraining and covered only classification. Our edits give a black-box behavioral analogue of exploitation for generative models, and we calibrate it with the same kind of duplication experiment.

On the QFS side, QMSum provides query-based meeting summarization over 232 meetings, with relevant-span annotations used to train its locator. It also reports that 74% of generated summaries contained factual inconsistencies, often misattributed speakers. QTSumm provides 7,111 query-summary pairs over Wikipedia tables. Both are the static, public benchmarks whose scores we propose to re-examine.

Research Program

The program has three parts.

Part 1: build edited benchmark variants. For QTSumm, we edit table cells that the gold summary depends on. These include swapping an entity, changing a number, or reordering a ranking. We regenerate the affected gold facts deterministically: re-sorting for ordering queries, recounting for counting queries, and substituting the value for lookups. Edits whose effect on the summary cannot be computed mechanically are discarded. For QMSum, we edit inside the annotated query-relevant spans. These edits change who said something, what was decided, or a stated figure, and we rewrite the corresponding clause of the reference summary. We also make matched edits of the same kind outside those spans, where the reference should not change. A random sample of edited pairs is checked by annotators to confirm the new reference is correct and the edited document stays coherent.

Each edit also gets a plausibility tag. Some edited values are as plausible as the original; others are implausible relative to world knowledge. This separates two confounds. A model might revert a Wikipedia table value because it knows the real-world fact (source recall). Alternatively, it might reproduce the benchmark's gold summary wording (gold recall). The two have different implications and are reported separately.

Part 2: define the measurements. Let $E$ be the set of query-relevant edits for a model. For each edit $e$ in $E$, let $o_e$ be the original value and $n_e$ the edited value. We define the leak rate as

$$L = \frac{1}{|E|}\sum_{e \in E} \mathbb{1}[\text{summary asserts } o_e],$$

and the grounding rate $G$ the same way, with $n_e$ in place of $o_e$. The remaining mass covers omission or other errors. Because the target values are known strings, entities, or numbers, checking assertion needs only entity and number matching, with a small LLM check on ambiguous cases. This avoids the 58.2% F1 judge problem that FABLES documents.

Focus is measured as the fraction of irrelevant edits under which the set of claims extracted from the summary is unchanged. Gold recall is measured as verbatim n-gram overlap with the original reference that the edited document no longer supports.

Part 3: calibrate the instrument and test one intervention. For calibration, we fine-tune an open model on the QTSumm and QMSum test pairs at duplication levels of 0, 1, 10, and 100. This follows the design of the memorization-versus-exploitation study. We check that leak rate rises monotonically with known contamination. Without this step, a high leak rate could just mean a model resists unusual inputs.

For the intervention, we fine-tune an open model on edited versions of the training splits only. This parallels DyVal's finding that training on generated data transfers. We then ask whether grounding improves on edited test documents without hurting original-benchmark scores. This tests whether the decomposed gap points to something trainable.

Baselines

The naive reference point is the original static benchmark score for each model, with no edits. This is what leaderboards report today. On QTSumm, the file reports GPT-4 zero-shot at 57.5 A3CU, OmniTab at 53.1, and OmniTab with REFACTOR at 54.0 (QTSumm). On QMSum, the strongest reported system is DYLE at 34.42 ROUGE-1, against DialogLM at 33.10 (DYLE). We rerun the strongest available reproductions under identical prompts and decoding on both original and edited sets: DYLE via its public code (Yale-LILY/DYLE), GPT-4-class models, and open models in the families named in the file (Llama-2-13B-chat, Mixtral, Qwen-72B). The proposal counts as informative relative to this baseline only if model rankings or gap sizes on the edited sets differ significantly from the original-set rankings, as judged by bootstrap intervals over documents.

The strongest existing method for contamination-free summarization evaluation is post-cutoff curation as in FABLES. We run it fairly with a small fresh QFS set: tables created after all tested models' cutoffs, with queries and summaries written by annotators following QTSumm's guidelines, sized to fit a budget comparable to FABLES. Our method beats this baseline if two conditions hold. First, edited-set scores predict model rankings on the fresh set at least as well as fresh-set annotation does on held-out fresh items. Second, they do so at a fraction of the annotation cost, while also giving the grounding, focus, and leak decomposition that fresh data alone cannot.

A second competitor is perturbation that preserves the answer: paraphrasing the query or reordering the document, in the spirit of robustness suites like PromptBench. It runs on the same documents and models.

For QFS systems, the strongest structured competitor is locate-then-summarize. This includes QMSum's locator pipeline, which raised BART ROUGE-1 from 29.20 to 31.74, and the modular TaPERA pipeline on QTSumm. For TaPERA, the file reports only relative gains of 5.4% in comprehensiveness and 5.0% in faithfulness over the second-best system; no reported absolute result was found on Lacuna. We include these pipelines because explicit retrieval should raise grounding. If they close most of the open-model gap on edited sets, that is itself a finding.

The key ablations isolate the two ingredients that make the probe work. One compares relevant against irrelevant edits, which separates grounding from focus. The other compares plausible against implausible edits, which separates benchmark recall from world-knowledge override.

Why not just…?

Why not just evaluate on fresh post-cutoff documents? This is the most serious alternative, and we include it as a baseline rather than dismiss it. Fresh data gives a clean absolute score. It is expensive, though: FABLES paid $5.2K for 26 books. It also ages quickly as models retrain. Most importantly, it cannot say why a model scores where it does on the benchmarks the field already reports. The edited sets reuse existing benchmarks, so they quantify how much each model's published score depended on recall. If the fresh-set comparison shows edited scores track fresh scores no better than original scores do, our method adds only the diagnostic decomposition, and we will report it as such.

Why not just paraphrase queries or shuffle documents? Paraphrasing keeps the correct answer the same, so a model that memorized the gold summary still scores perfectly. Perturbation that preserves the answer measures surface robustness, not reading. The direct comparison between paraphrase and fact-edit sets will show this. We predict paraphrase scores stay close to original scores while fact-edit scores drop, especially in the duplication-calibrated models.

Why not just run a membership-inference or n-gram overlap detector? Such detectors need model probabilities or access to training corpora, which proprietary APIs do not expose. That is the very asymmetry that makes open-versus-proprietary comparisons hard. The memorization-versus-exploitation study also shows that detecting memorization does not establish that it changes task scores. Our probe measures the behavioral effect directly and needs only generated text.

Why not just generate fully synthetic documents, as DyVal does? Synthetic meetings would lack the speaker dynamics and scattered evidence that make QMSum hard. DyVal itself notes it could not extend its generator to such text. Editing real documents keeps the natural distribution and changes only what we need to control.

Decisive Checks

Four results would settle the claim.

First, the duplication-calibrated open model must show leak rate rising with duplication count. If it does not, the probe is not measuring contamination, and the project stops at that negative result.

Second, compare the open-versus-proprietary gap on edited sets with the gap on original sets for QTSumm and QMSum. A significant widening supports the prediction drawn from INVITE. A significant narrowing would mean proprietary scores were the more recall-inflated ones.

Third, look at the plausible-versus-implausible split. Leakage concentrated on implausible edits in Wikipedia tables but absent in QMSum meetings would point to world knowledge rather than benchmark contamination.

Fourth, fine-tuning on edited training splits must raise grounding on edited test documents without lowering original A3CU or ROUGE scores. That would show the gap component we measure can be reduced by training.