KEEP-Bench: Evaluating Evidence Preservation in Context Compression

Ziqing Man, Di Wu, Jia-Chen Gu, Junyi Zhang, Kai-Wei Chang, Nanyun PengarXiv 2026 · arXiv

TL;DR

KEEP-Bench is a new diagnostic benchmark designed to expose "faithfulness failures" in context compression that traditional Q&A metrics often miss. By evaluating whether compressed text retains every essential atomic proposition, the authors reveal that even state-of-the-art models like Gemma-4-31B and Claude-Haiku-4.5 fail to preserve critical evidence in nearly 40% of cases, even when the input is well within their context window.

What Problem Does This Paper Address?

As Large Language Models (LLMs) tackle increasingly long documents, "context compression"—reducing the input length to save on inference costs—has become essential. However, compression creates an information bottleneck. If a compressor omits a subtle but vital piece of evidence, the "reader" model might hallucinate or fail.

The authors argue that current evaluation methods are flawed. Most researchers judge a compressor by whether a model can still answer a question correctly after seeing the compressed text. But as shown in Figure 1, this is a misleading proxy: a model might get the right answer using its own internal knowledge or by taking "shortcuts," even if the compressed context is unfaithful or factually broken.

Figure 1: Comparison of indirect evaluation and direct evidence evaluation on a simplified KEEP-Bench case. The upper branch shows indirect evaluation through downstream QA, which considers only the final answer and may therefore falsely accept an unfaithful compressed context. The lower branch shows KEEP-Bench’s direct evidence evaluation, which independently checks whether each reference evidence proposition is entailed by the compressed context, thereby detecting the preservation failure.

Methods

KEEP-Bench moves away from "answer accuracy" and toward Direct Evidence Evaluation. The core idea is to decompose the necessary evidence for a query into a set of minimal, self-contained propositions. A compressor is only successful if the resulting summary entails every single proposition (a metric called All-Rec).

The benchmark introduces three specific "stress tests" for evidence structures:

  1. Parallel Evidence: Uses underspecified entities (e.g., two different people with the same last name) to force the model to track two separate reasoning chains simultaneously.
  2. Conditional Evidence: Requires the model to keep both a fact and the specific "qualifier" or condition that makes it true.
  3. Imbalanced Evidence: Mixes "dominant" perspectives (supported by lots of text) with "minority" perspectives (supported by only a single sentence) to see if the compressor ignores the rare but necessary info.

To measure efficiency, the authors introduce Evidence Preservation Return (EPR), which calculates how much evidence recall a model provides per unit of computational effort (TFLOPs).

Datasets

The authors introduced KEEP-Bench, which contains 4,102 cases.

  • Parallel and Conditional settings are built using fictionalized versions of the MuSiQue dataset to ensure models can't rely on memorized world knowledge.
  • Imbalanced settings use real-world scientific papers and policy documents (e.g., gift card laws across different countries).

Key Insights and Ideas

  • The "Faithfulness Gap": In a pilot study, the authors found that while models could answer questions with 96% accuracy on "faithful" compressions, accuracy plummeted to 11.5% on "unfaithful" ones. Traditional metrics often fail to detect this because they don't isolate the compressor's performance from the reader's behavior.
  • Scaling has Diminishing Returns: While larger models generally preserve more evidence, the EPR metric shows that efficiency actually peaks at moderate scales (4B–8B parameters). Beyond that, the massive increase in FLOPs doesn't yield a proportional increase in evidence recall.
  • Primacy Bias: Evidence appearing earlier in a document is significantly more likely to be preserved than evidence appearing later, suggesting compressors have a "first-come, first-served" bias.
  • Structure Recognition is Key: Increasing the token budget for a summary helps strong models get even better, but it does almost nothing for weak models. If a model doesn't "recognize" the underlying logical structure (like a minority perspective), giving it more words to write won't help it find what it already missed.

Key Results

The results are a wake-up call for the field. Despite source contexts averaging only 3.1K tokens—well within the 100K+ limits of modern models—preservation is surprisingly poor:

  • Best Open-Weight Model: Gemma-4-31B-it achieved only 66.8% All-Rec.
  • Best Proprietary Model: Claude-Haiku-4.5 reached 61.6% All-Rec.
  • Dedicated Compressors: Specialized, smaller compression models (like RECOMP or EXIT) performed poorly, often staying below 10% All-Rec.

As seen in Figure 3, there is a massive "empty zone" in the upper-left quadrant: no current method provides high preservation at a low computational cost.

Figure 3: Computation cost and evidence preservation under the default compression budget. Overall All-Rec versus av- erage TFLOPs per case on a log scale. Circle area represents model size; the dashed line marks the Pareto frontier.

Limitations and Future Work

The authors acknowledge that their study focuses on textual compression; they did not evaluate "latent" compression (where context is compressed into KV-cache embeddings).

The primary takeaway is that we need a new generation of compressors. Future work should focus on models that can recognize complex evidence structures (like qualifiers and parallel chains) and preserve them efficiently, rather than simply scaling up model size or summary length. The goal is to move the "preservation frontier" toward the upper-left of the cost-benefit graph.

Summary updated

Abstract

Context compression improves long-context efficiency but may introduce faithfulness issues through an information bottleneck. Existing evaluations rely on QA metrics, but a reader can still answer correctly from an unfaithful compression, so these metrics underestimate such loss. To bridge this gap, we introduce KEEP-Bench, a diagnostic benchmark that directly assesses compression faithfulness with 4,102 cases across three challenging settings that require keeping evidence for plausible reference resolutions, explicit qualifiers, and minority perspectives. Experiments show that even with tailored prompting, the strongest models remain unreliable: despite context windows exceeding 100K tokens and inputs averaging only 3.1K tokens, the best open-weight model (31B) preserves all required evidence in only 66.8% of cases, and the best proprietary model in only 61.6%. Performance also varies sharply across evidence structures, so conclusions from limited structures may be biased. We further introduce a metric, Evidence Preservation Return (EPR), to relate evidence recall to inference FLOPs, and find that it peaks at moderate scales (4B-8B), where recall remains low. Additional output budget mainly benefits already strong models and widens capability gaps, yet the stronger preservation of these models comes at a steep computational cost. These findings highlight the importance for future work to investigate compressors with strong information preservation under a small computation budget, rather than relying on a large output budget or model scaling.
BibTeX
@misc{man2026keep,
  author = {Ziqing Man and Di Wu and Jia-Chen Gu and Junyi Zhang and Kai-Wei Chang and Nanyun Peng},
  title = {{KEEP-Bench: Evaluating Evidence Preservation in Context Compression}},
  year = {2026},
  eprint = {2610.26827},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26827},
}
Figure