TL;DR
KEEP-Bench is a new diagnostic benchmark designed to expose "faithfulness failures" in context compression that traditional Q&A metrics often miss. By evaluating whether compressed text retains every essential atomic proposition, the authors reveal that even state-of-the-art models like Gemma-4-31B and Claude-Haiku-4.5 fail to preserve critical evidence in nearly 40% of cases, even when the input is well within their context window.
What Problem Does This Paper Address?
As Large Language Models (LLMs) tackle increasingly long documents, "context compression"—reducing the input length to save on inference costs—has become essential. However, compression creates an information bottleneck. If a compressor omits a subtle but vital piece of evidence, the "reader" model might hallucinate or fail.
The authors argue that current evaluation methods are flawed. Most researchers judge a compressor by whether a model can still answer a question correctly after seeing the compressed text. But as shown in Figure 1, this is a misleading proxy: a model might get the right answer using its own internal knowledge or by taking "shortcuts," even if the compressed context is unfaithful or factually broken.
Methods
KEEP-Bench moves away from "answer accuracy" and toward Direct Evidence Evaluation. The core idea is to decompose the necessary evidence for a query into a set of minimal, self-contained propositions. A compressor is only successful if the resulting summary entails every single proposition (a metric called All-Rec).
The benchmark introduces three specific "stress tests" for evidence structures:
- Parallel Evidence: Uses underspecified entities (e.g., two different people with the same last name) to force the model to track two separate reasoning chains simultaneously.
- Conditional Evidence: Requires the model to keep both a fact and the specific "qualifier" or condition that makes it true.
- Imbalanced Evidence: Mixes "dominant" perspectives (supported by lots of text) with "minority" perspectives (supported by only a single sentence) to see if the compressor ignores the rare but necessary info.
To measure efficiency, the authors introduce Evidence Preservation Return (EPR), which calculates how much evidence recall a model provides per unit of computational effort (TFLOPs).
Datasets
The authors introduced KEEP-Bench, which contains 4,102 cases.
- Parallel and Conditional settings are built using fictionalized versions of the MuSiQue dataset to ensure models can't rely on memorized world knowledge.
- Imbalanced settings use real-world scientific papers and policy documents (e.g., gift card laws across different countries).
Key Insights and Ideas
- The "Faithfulness Gap": In a pilot study, the authors found that while models could answer questions with 96% accuracy on "faithful" compressions, accuracy plummeted to 11.5% on "unfaithful" ones. Traditional metrics often fail to detect this because they don't isolate the compressor's performance from the reader's behavior.
- Scaling has Diminishing Returns: While larger models generally preserve more evidence, the EPR metric shows that efficiency actually peaks at moderate scales (4B–8B parameters). Beyond that, the massive increase in FLOPs doesn't yield a proportional increase in evidence recall.
- Primacy Bias: Evidence appearing earlier in a document is significantly more likely to be preserved than evidence appearing later, suggesting compressors have a "first-come, first-served" bias.
- Structure Recognition is Key: Increasing the token budget for a summary helps strong models get even better, but it does almost nothing for weak models. If a model doesn't "recognize" the underlying logical structure (like a minority perspective), giving it more words to write won't help it find what it already missed.
Key Results
The results are a wake-up call for the field. Despite source contexts averaging only 3.1K tokens—well within the 100K+ limits of modern models—preservation is surprisingly poor:
- Best Open-Weight Model: Gemma-4-31B-it achieved only 66.8% All-Rec.
- Best Proprietary Model: Claude-Haiku-4.5 reached 61.6% All-Rec.
- Dedicated Compressors: Specialized, smaller compression models (like RECOMP or EXIT) performed poorly, often staying below 10% All-Rec.
As seen in Figure 3, there is a massive "empty zone" in the upper-left quadrant: no current method provides high preservation at a low computational cost.
Limitations and Future Work
The authors acknowledge that their study focuses on textual compression; they did not evaluate "latent" compression (where context is compressed into KV-cache embeddings).
The primary takeaway is that we need a new generation of compressors. Future work should focus on models that can recognize complex evidence structures (like qualifiers and parallel chains) and preserve them efficiently, rather than simply scaling up model size or summary length. The goal is to move the "preservation frontier" toward the upper-left of the cost-benefit graph.
Summary updated