CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets

Huzaifa Pardawala, Siddhant Sukhani, Sudheer Chava, Agam ShaharXiv 2026 · arXiv

TL;DR

CoRA-Bench is a new evaluation framework designed to measure how effectively context selection methods—like retrieval and compression—preserve the "needles" of evidence required to answer queries within strict token budgets. The benchmark reveals that there is no universal "best" selector; rather, performance is highly task-dependent, with lexical methods excelling at explicit fact retrieval while dense and compression-based methods dominate in semantic and conversational tasks.

What Problem Does This Paper Address?

As Large Language Models (LLMs) support increasingly massive context windows, the cost and latency of processing millions of tokens become significant bottlenecks. Context selection (reducing the input to only the most relevant parts) is a common solution, but it introduces a dangerous risk: discarding the specific evidence needed to answer a query.

Existing evaluations often fail to compare different types of selectors (e.g., traditional search vs. neural compression) under identical constraints. Furthermore, high "retrieval recall" doesn't always translate to high "downstream accuracy." CoRA-Bench addresses this by providing a unified protocol to evaluate evidence preservation, token retention, and reader accuracy across diverse long-context regimes under shared token budgets (32K, 64K, and 128K).

Methods

The authors standardized a four-stage evaluation pipeline:

  1. Standardize: Source text is broken into 512-token chunks with a 64-token overlap.
  2. Select: Six practical selectors are compared:
  3. Positional: Head and tail truncation (a simple baseline).
  4. Lexical: BM25 (traditional keyword-based search).
  5. Dense Retrieval: Using Qwen3-Embedding-0.6B.
  6. Reranking: Dense retrieval followed by Qwen3-Reranker-0.6B.
  7. Compression: LLMLingua-2 (query-agnostic token pruning) and Provence (query-guided sentence-level pruning).
  8. Merge + Budget: Selected chunks are merged to remove overlaps and capped at the token budget (e.g., 32K).
  9. Score: Evidence preservation is measured via recall, and downstream performance is tested using a fixed reader model (Gemini 2.5 Flash-Lite).
Figure 1: CoRA-Bench: shared token budgets and dataset-specific evidence scoring across four retrieval regimes. Evidence and reader evaluation use the same final context, with overlapping retrieval spans merged; reader inference is repeated with 3 seeds. The LM-filter bypass denotes Threshold ≥50, uncapped; its ranked, capped arms follow budget enforcement without a score threshold.

Datasets

The benchmark spans four complementary long-context settings:

  • BABILong: Tests retrieval of sparse supporting facts hidden in long book texts.
  • LOONG Financial Spotlight: Focuses on evidence localization within multi-document financial reports.
  • NoLiMa: A custom diagnostic designed to test retrieval where there is minimal literal (keyword) overlap between the query and the evidence.
  • LongMemEval: Evaluates conversational retrieval from long, multi-session dialogue histories.
Table 1: Evaluated data. Lengths use cl100k_base. NoLiMa is a custom repeated-measures diagnostic, not the official benchmark aggregate. LongMemEval is treated as static context selection, not online memory management.

Key Insights and Ideas

  • Lexical vs. Semantic Strengths: BM25 is surprisingly resilient. It leads in evidence recall on BABILong because that task relies on explicit surface cues. However, on NoLiMa—where the "semantic gap" is intentionally wide—BM25 fails, and dense/compression methods take the lead.
  • The Recall-Accuracy Gap: High evidence recall is a prerequisite for accuracy, but not a guarantee. Interestingly, for text-modifying methods like Provence, the authors found that reader accuracy can remain high even when verbatim evidence recall is lower, suggesting these methods preserve semantic utility even if they alter the exact wording.
  • Selection is Better than "Full Context": In several cases, providing a selected 32K context resulted in higher accuracy than providing the entire context. This suggests that "noise" in massive contexts can actively confuse even long-context-capable models.
  • Compute Trade-offs: There is a massive variance in the "cost of selection." While BM25 is nearly free (CPU-only), Provence is computationally expensive, requiring significantly more GPU hours to prune context effectively.

Key Results

  • BABILong (32K Budget): BM25 achieved 46.20% accuracy and Provence reached 48.13%, both significantly outperforming the "Full Context" baseline of 31.13%.
  • NoLiMa (Semantic Retrieval): Provence was the standout performer, achieving 27.43% accuracy, whereas most other methods (including BM25 and Head/Tail) hovered near 0%.
  • LongMemEval (Conversational): Dense retrieval proved most effective, reaching 71.13% accuracy at a 32K budget.
  • Efficiency: Dense retrieval provided a strong balance of performance and speed, taking only 5.1 L4 instance-hours compared to Provence’s 25.6 hours.
Table 2: Reader accuracy (%) at a 32K context budget under overlap-union packing, averaged over seeds 5768, 78516, and 944601. LOONG uses Perfect Rate; NoLiMa is the custom diagnostic. Full and empty controls use all questions without a context budget. Confidence intervals and all budgets are reported in Appendix B.

Limitations and Future Work

The authors acknowledge several limitations:

  • Reader Dependency: Results are conditional on the specific reader model used (Gemini 2.5 Flash-Lite). Different LLMs might respond differently to the same selected context.
  • Verbatim Scoring: The evidence recall metric requires verbatim survival of text. This may unfairly penalize compression methods that successfully preserve the meaning of a fact but rewrite the sentence.
  • Static vs. Dynamic: The current benchmark treats context selection as a static task. Future work could extend this to "online" memory systems that must update and prune information dynamically over many interactions.
Table 3: Historical per-method selection compute under chunk-concatenation packing, in L4 instance wall-clock hours (recorded via time.perf_counter around each selector’s main compute block; this is instance wall time, not measured GPU utilization). CPU-only baselines and API-based LM filtering are excluded. LM-filter API costs are reported separately in the text.

Summary updated

Abstract

Context selection can reduce input length during long-input inference but can remove evidence required to answer a query. We introduce CoRA-Bench, a benchmark for evaluating evidence preservation under shared token budgets. We compare head and tail truncation, BM25, dense retrieval, dense retrieval with reranking, LLMLingua-2, and Provence across BABILong, LOONG Financial Spotlight, a custom NoLiMa diagnostic, and LongMemEval. Evidence-preservation results reveal task-dependent tradeoffs: BM25 achieves 0.921 supporting-fact recall on BABILong at 128K, dense retrieval and dense reranking each achieve 0.908 needle recall on NoLiMa at 64K, and dense retrieval achieves 0.993 answer-bearing-turn recall on LongMemEval at 64K. Evidence, token retention, and complementary reader evaluations use the same selected contexts. Reader evaluations use dataset-native answer metrics and three runs on the same questions, with paired bootstrap confidence intervals. At a 32K budget, BM25 achieves 46.20% BABILong answer accuracy and dense retrieval achieves 71.13% LongMemEval accuracy. The results show that method rankings vary across evidence types and between evidence recall and downstream accuracy.
BibTeX
@misc{pardawala2026cora,
  author = {Huzaifa Pardawala and Siddhant Sukhani and Sudheer Chava and Agam Shah},
  title = {{CoRA-Bench: Evaluating Evidence Preservation by Context Selectors Under Token Budgets}},
  year = {2026},
  eprint = {2611.10238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10238},
}
Figure