TL;DR
CoRA-Bench is a new evaluation framework designed to measure how effectively context selection methods—like retrieval and compression—preserve the "needles" of evidence required to answer queries within strict token budgets. The benchmark reveals that there is no universal "best" selector; rather, performance is highly task-dependent, with lexical methods excelling at explicit fact retrieval while dense and compression-based methods dominate in semantic and conversational tasks.
What Problem Does This Paper Address?
As Large Language Models (LLMs) support increasingly massive context windows, the cost and latency of processing millions of tokens become significant bottlenecks. Context selection (reducing the input to only the most relevant parts) is a common solution, but it introduces a dangerous risk: discarding the specific evidence needed to answer a query.
Existing evaluations often fail to compare different types of selectors (e.g., traditional search vs. neural compression) under identical constraints. Furthermore, high "retrieval recall" doesn't always translate to high "downstream accuracy." CoRA-Bench addresses this by providing a unified protocol to evaluate evidence preservation, token retention, and reader accuracy across diverse long-context regimes under shared token budgets (32K, 64K, and 128K).
Methods
The authors standardized a four-stage evaluation pipeline:
- Standardize: Source text is broken into 512-token chunks with a 64-token overlap.
- Select: Six practical selectors are compared:
- Positional: Head and tail truncation (a simple baseline).
- Lexical: BM25 (traditional keyword-based search).
- Dense Retrieval: Using Qwen3-Embedding-0.6B.
- Reranking: Dense retrieval followed by Qwen3-Reranker-0.6B.
- Compression: LLMLingua-2 (query-agnostic token pruning) and Provence (query-guided sentence-level pruning).
- Merge + Budget: Selected chunks are merged to remove overlaps and capped at the token budget (e.g., 32K).
- Score: Evidence preservation is measured via recall, and downstream performance is tested using a fixed reader model (Gemini 2.5 Flash-Lite).
Datasets
The benchmark spans four complementary long-context settings:
- BABILong: Tests retrieval of sparse supporting facts hidden in long book texts.
- LOONG Financial Spotlight: Focuses on evidence localization within multi-document financial reports.
- NoLiMa: A custom diagnostic designed to test retrieval where there is minimal literal (keyword) overlap between the query and the evidence.
- LongMemEval: Evaluates conversational retrieval from long, multi-session dialogue histories.
Key Insights and Ideas
- Lexical vs. Semantic Strengths: BM25 is surprisingly resilient. It leads in evidence recall on BABILong because that task relies on explicit surface cues. However, on NoLiMa—where the "semantic gap" is intentionally wide—BM25 fails, and dense/compression methods take the lead.
- The Recall-Accuracy Gap: High evidence recall is a prerequisite for accuracy, but not a guarantee. Interestingly, for text-modifying methods like Provence, the authors found that reader accuracy can remain high even when verbatim evidence recall is lower, suggesting these methods preserve semantic utility even if they alter the exact wording.
- Selection is Better than "Full Context": In several cases, providing a selected 32K context resulted in higher accuracy than providing the entire context. This suggests that "noise" in massive contexts can actively confuse even long-context-capable models.
- Compute Trade-offs: There is a massive variance in the "cost of selection." While BM25 is nearly free (CPU-only), Provence is computationally expensive, requiring significantly more GPU hours to prune context effectively.
Key Results
- BABILong (32K Budget): BM25 achieved 46.20% accuracy and Provence reached 48.13%, both significantly outperforming the "Full Context" baseline of 31.13%.
- NoLiMa (Semantic Retrieval): Provence was the standout performer, achieving 27.43% accuracy, whereas most other methods (including BM25 and Head/Tail) hovered near 0%.
- LongMemEval (Conversational): Dense retrieval proved most effective, reaching 71.13% accuracy at a 32K budget.
- Efficiency: Dense retrieval provided a strong balance of performance and speed, taking only 5.1 L4 instance-hours compared to Provence’s 25.6 hours.
Limitations and Future Work
The authors acknowledge several limitations:
- Reader Dependency: Results are conditional on the specific reader model used (Gemini 2.5 Flash-Lite). Different LLMs might respond differently to the same selected context.
- Verbatim Scoring: The evidence recall metric requires verbatim survival of text. This may unfairly penalize compression methods that successfully preserve the meaning of a fact but rewrite the sentence.
- Static vs. Dynamic: The current benchmark treats context selection as a static task. Future work could extend this to "online" memory systems that must update and prune information dynamically over many interactions.
Summary updated