You Only Use Reactive Attention Slice When Retrieving From Long Context

Yun Joon Soh, Hanxian Huang, Yuandong Tian, Jishen ZhaoEMNLP 2025

TL;DR

This paper introduces YOURA, a novel, training-free retrieval method for Large Language Models (LLMs) that significantly improves performance on long-context tasks. By analyzing how an LLM's self-attention "reacts" to a query, YOURA efficiently identifies and retrieves only the most relevant sentences, leading to better accuracy and faster inference.

What Problem Does This Paper Address?

LLMs are powerful, but their ability to process long texts is limited by context window size. While Retrieval-Augmented Generation (RAG) helps by retrieving relevant information, existing methods have drawbacks. Embedding-based retrieval often struggles with semantic nuances and can be misled by superficial word overlap. Neural retrievers, while more sophisticated, are computationally expensive to train. This paper tackles the challenge of efficient and accurate retrieval for long contexts, aiming to bridge the gap between LLM capabilities and the need to process extensive information without prohibitive computational costs.

Methods

The core of YOURA is a novel attention-based retrieval technique that doesn't require training or fine-tuning. It works by:

  • Reaction Vector Calculation: YOURA first computes the LLM's self-attention on the context without the query. Then, it computes the self-attention with the query appended. The "reaction vector" is derived by comparing these two attention patterns, highlighting how the LLM's attention shifts in response to the query. This vector quantifies the importance of each token in the context relative to the query.
  • Sentence Extraction Algorithm: To enable granular retrieval, the paper proposes an algorithm to accurately split the token sequence into sentences, achieving 94% accuracy.
  • Reaction Score: The reaction vector is then sliced by sentence, and a "reaction score" (geometric mean of the vector slice) is calculated for each sentence. This score represents the semantic relevance of the sentence to the query.
  • Retrieval: Sentences are greedily retrieved based on their reaction scores until a token budget is met.

Datasets

The paper evaluates YOURA on several datasets:

  • LongBench: A benchmark for long-context understanding, including single-document QA, multi-document QA, summarization, and few-shot learning tasks.
  • BABILong: Used for Needle-In-A-Haystack style evaluation, specifically for QA tasks (QA1-QA5).
  • Additional datasets for summarization, few-shot learning, and synthetic tasks are mentioned in the appendix.

Key Insights and Ideas

  • Attention as a "Reaction" Indicator: The key insight is that the change in an LLM's self-attention when a query is introduced is a powerful signal of semantic relevance, more so than just static attention weights.
  • Holistic Contextual Relevance: YOURA's approach of analyzing attention shifts across the entire context, rather than in isolation for each chunk, allows it to capture inter-chunk dependencies and provide more nuanced retrieval.
  • Training-Free Efficiency: The ability to achieve strong results without any training or fine-tuning is a significant advantage, making YOURA easily adaptable to various LLMs.
  • Impact of Retrieved Context Size: The experiments highlight that LLM serving platforms are highly sensitive to the size of the retrieved context, underscoring the importance of efficient and accurate retrieval.

Key Results

  • Improved QA Accuracy: YOURA improves QA task accuracy by up to 15% compared to embedding-based retrieval on the LongBench dataset.
  • Increased Inference Throughput: YOURA achieves up to 31% higher inference throughput compared to embedding-based retrieval, primarily because it retrieves fewer tokens.
  • Competitive Performance: On the BABILong dataset, YOURA improved Llama3's accuracy by 25%.
  • Robustness: The sentence extraction algorithm demonstrates resilience to typos and variations in tokenizers.

Limitations and Future Work

  • Approximation of Attention Vector: The current calculation of the reaction vector is an approximation, and future work could explore more precise methods, especially for multi-head attention.
  • Attention as Importance: The paper acknowledges the ongoing debate about whether attention weights truly reflect ground-truth importance, though it finds the heuristic effective in practice.
  • Scalability: While YOURA can process partitioned inputs, optimizing GPU memory management for parallel processing of these partitions is a future direction.
  • Overlapping Attention: Determining reaction scores for overlapping sentences in alternative attention mechanisms is left for future exploration.
  • Bias and Fairness: The method might inadvertently amplify biases present in the pre-trained LLM. Future work could explore bias correction techniques.
  • Privacy: Preprocessed data caching poses potential privacy risks, and robust anonymization is recommended.
  • Language Dependence: YOURA's effectiveness might be language-dependent, requiring future research into cross-linguistic dataset inclusivity.

Summary updated

BibTeX
@inproceedings{soh2025you,
  author = {Yun Joon Soh and Hanxian Huang and Yuandong Tian and Jishen Zhao},
  title = {{You Only Use Reactive Attention Slice When Retrieving From Long Context}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure