Reasoning-Focused Evaluation of Efficient Long-Context Inference Techniques

Joie Zhang, Qiyao Wei, Howard Yen, Xi Ye, Danqi ChenER @ NeurIPS 2025 · OpenReview

TL;DR

This paper investigates efficient inference techniques for large language models (LLMs) on tasks requiring long-context reasoning, focusing on both input processing and output generation. They find that weight quantization (NF4, Int8) offers a superior Pareto-optimal balance of performance and memory compared to KV cache eviction methods, which struggle with tasks demanding precise string retrieval. Notably, reasoning-focused LLMs show promise in mitigating performance degradation caused by these eviction techniques.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown remarkable abilities in complex reasoning, often by generating "chains of thought" (CoT) that leverage information spread across extensive contexts. However, this capability comes at a significant computational cost, primarily due to the explosion in the size of the KV cache, which impacts memory usage, operational costs, and scalability. Existing research on efficient inference techniques for long contexts has largely focused on maintaining performance for long inputs. This paper argues that this focus is insufficient, as it overlooks two critical aspects of reasoning models: their ability to generate long outputs and their capacity to synthesize dispersed information from both the input and previously generated outputs. The authors aim to bridge this gap by systematically evaluating various efficiency techniques on tasks that specifically challenge these reasoning capabilities, moving beyond simple retrieval tasks.

Methods

The study evaluates six efficient inference techniques:

  • Weight Quantization:
    • NF4 (4-bit NormalFloat): An information-theoretically optimal quantization data type for normally distributed weights, aiming for high compression with minimal performance loss.
  • Int8 (8-bit quantization): A standard technique for reducing model size and computational cost.
  • KV Cache Token Eviction: These methods aim to reduce KV cache memory by selectively discarding tokens.
  • DuoAttention: A state-of-the-art token eviction method.
  • SnapKV, PyramidKV, StreamingLLM: Other canonical token eviction techniques.

The evaluation is conducted on both instruction-tuned and reasoning-focused LLMs.

Datasets

The paper uses a combination of tasks from two benchmarks:

  • HELMET: A broad benchmark covering tasks like recall (NIAH, JSON KV), retrieval (HotpotQA, Natural Questions), citation generation (ASQA Cite), document reranking (MS MARCO), in-context learning (BANKING77, CLINC150), and summarization (Multi-LexSum). These tasks are evaluated at 16K and 32K context lengths.
  • LongProc: This benchmark focuses on tasks requiring long-output generation and dispersed information synthesis, including HTML-to-TSV, Pseudocode-to-Code, and Travel Planning. These are evaluated at 0.5K and 2K output lengths.

The authors emphasize that these tasks are chosen to represent varying levels of information dispersion and output generation length, moving beyond simple "needle-in-a-haystack" scenarios.

Key Insights and Ideas

  • Quantization Reigns Supreme (Pareto-Optimal): NF4 and Int8 quantization consistently outperform KV cache eviction methods in terms of the Pareto frontier (balancing performance and memory usage) across most models and tasks. This suggests that compressing model weights is a more robust and efficient strategy for long-context reasoning than dynamically managing the KV cache.
  • Token Eviction's Achilles' Heel: String Retrieval: Token eviction methods significantly degrade on tasks requiring precise string retrieval (e.g., identifying specific IDs or keywords). The hypothesis is that these methods tend to discard critical tokens necessary for exact matching, leading to failure.
  • Reasoning Models as Resilience Boosters: Reasoning-focused LLMs demonstrate a remarkable ability to mitigate the performance degradation caused by token eviction techniques, especially on in-context learning (ICL) tasks. They can achieve near-baseline performance even with aggressive eviction, suggesting their internal reasoning structure is more robust to token loss.
  • Task Difficulty Matters: Performance degradation is not uniform. Tasks with long outputs and high information dispersion are particularly challenging for current efficient inference methods, highlighting the need for more comprehensive evaluation benchmarks.

Key Results

  • Quantization Efficiency: NF4 and Int8 quantization achieve substantial memory savings (e.g., ~48% for NF4) with minimal performance degradation (e.g., ~2.5% for NF4).
  • Token Eviction Struggles: Token eviction methods (SnapKV, PyramidKV, StreamingLLM) show significant performance degradation (averaging ~23%) and often have memory overhead when trying to maintain performance. DuoAttention is an exception, showing some performance improvement but still struggling on harder tasks.
  • Task Difficulty Impact: The baseline model's performance drops by 58.4% from easy (short output, low dispersion) to hard (long output, high dispersion) tasks. Quantization methods maintain near-baseline performance across all difficulties, while token eviction methods degrade substantially.
  • Reasoning Model Resilience: On ICL tasks, a reasoning model (DeepSeek-R1-Distill-Llama-8B) showed only a 0.8% performance drop with aggressive eviction (SnapKV, window=256, cache=2048), compared to a 5.7% drop for an instruct model (Llama-3.1-8B-Instruct) under the same conditions.

Limitations and Future Work

  • Model Scale: The study is primarily conducted on 7B and 8B parameter models. The authors acknowledge that the dynamics of CoT verbosity, KV cache explosion, and efficiency technique effectiveness might differ significantly in much larger models (70B+).
  • Task Specificity: While the task selection is broad, it's not exhaustive. The observed benefits of reasoning models in mitigating token eviction degradation were most pronounced on ICL tasks and did not extend to tasks like document reranking, which are highly sensitive to information loss.
  • Future Directions:
    • Hybrid Methods: Combining quantization with more sophisticated, content-aware KV cache eviction strategies to further push the Pareto frontier.
  • Adaptive Inference: Developing techniques that dynamically adjust cache management policies based on the inferred task type.
  • Wider Model Evaluation: Extending the evaluation to a broader range of models, including larger ones, to understand how efficiency techniques interact with different model capabilities.

Summary updated

Abstract

The rise of large reasoning models has demonstrated impressive capabilities in tackling increasingly complex tasks that often require generating long chains of thought. However, such capability also comes with a substantial increase in inference cost. Despite this growing trend, existing evaluations of efficient long-context inference techniques have primarily focused on preserving model performance for long inputs. We argue that this focus overlooks two crucial aspects of reasoning models: (1) their ability to generate long outputs, and (2) their ability to synthesize dispersed information -- to reason over information distributed across both the input context and previously generated outputs. In this work, we systematically evaluate an array of efficient inference techniques across 12 tasks that differ in their requirements for long-output generation and information synthesis, on both instruction-tuned and reasoning models. We find that NF4 and Int8 weight quantization strongly outperform baselines and key-value (KV) cache token eviction methods on Pareto-optimality in both memory and performance across evaluated models, tasks, and context lengths. We observe that existing long-context efficient inference techniques perform especially poorly on tasks with long outputs and high token dispersion, which are critical for reasoning. Specifically, our experiments suggest that token eviction methods struggle in these settings because they cannot reliably perform exact string retrieval, likely because they tend to eject critical tokens during decoding. Finally, we find that using reasoning models can partially mitigate this degradation, recovering performance close to the full-cache baseline even at smaller cache sizes, thereby advancing the Pareto frontier of memory efficiency and long-context performance.
BibTeX
@inproceedings{zhang2025reasoning,
  author = {Joie Zhang and Qiyao Wei and Howard Yen and Xi Ye and Danqi Chen},
  title = {{Reasoning-Focused Evaluation of Efficient Long-Context Inference Techniques}},
  booktitle = {ER @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=uCOMb0EsPq},
}
Figure