Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs

Yunlong Zhao, Haoran Wu, Bo XuAAAI 2025 · DOI

Abstract

Processing long-context prompts in Large Language Models (LLMs) presents significant computational challenges and can lead to performance degradation. Traditional prompt compression methods often rely on information entropy (perplexity), which may not align with a model's actual information needs for a specific query. This paper introduces AttnComp, a novel compression framework that utilizes an LLM's internal attention mechanisms. By extracting query-guided cross-attention to rank token importance and self-attention to identify cohesive "semantic units," AttnComp achieves higher compression ratios with superior accuracy and lower latency compared to existing state-of-the-art methods.

Introduction

As LLMs are increasingly used for tasks involving massive contexts—such as analyzing entire books or large document repositories—they encounter a "computational wall." Long prompts are not only expensive and slow to process but can also trigger the "lost-in-the-middle" phenomenon, where model performance drops as context length increases.

Existing prompt compression tools typically use perplexity as a proxy for token importance. However, this approach has two primary drawbacks: it ignores the specific query context and assumes token independence, often resulting in fragmented, ungrammatical compressed prompts that confuse the target model. AttnComp addresses these issues by leveraging the "intrinsic wisdom" of the model's own attention layers to perform query-aware, semantically-preserved compression.

Methodology: The AttnComp Framework

AttnComp operates through a three-stage pipeline designed to identify and preserve the most relevant information within a prompt.

The AttnComp architecture: From raw context and query to a compressed prompt using semantic unit identification and attention-based filtering.

1. Attention Extraction

A small, efficient "helper" model (e.g., Llama-2-7B) processes the original prompt to extract two critical data types:

  • Query-guided Cross-Attention (CA): This identifies which parts of the context the query specifically attends to.
  • Self-Attention (SA): This captures the relationships between context tokens, revealing the underlying structure of the text.

2. Semantic Unit Identification

To avoid the pitfalls of token-level pruning, AttnComp groups tokens into "semantic units" (phrases or clauses). The prompt is modeled as a graph where tokens are nodes and self-attention weights serve as edges. A Maximum Spanning Tree (MST) algorithm identifies the strongest semantic connections, followed by the Louvain algorithm for community detection to cluster tokens into units that should be treated as single entities during compression.

3. Dynamic Filtering

Semantic units are ranked based on their cross-attention scores. The system then filters out the lowest-scoring units to reach the desired compression ratio. This ensures that the remaining text is both query-relevant and semantically coherent.

Experimental Evaluation

The researchers evaluated AttnComp against major benchmarks, including Natural Questions (NQ) for retrieval-augmented generation and LongBench for various long-context tasks like single-document QA and multi-document reasoning.

Key Results and Insights

AttnComp demonstrates significant improvements over perplexity-based baselines like LLMLingua and LongLLMLingua. On the NQ dataset using GPT-3.5-Turbo, AttnComp achieved 76.8% accuracy at a 4.5x compression ratio, notably outperforming the uncompressed original prompt in certain configurations (75.7%).

Accuracy vs. filtration percentage: AttnComp (Our Cross-Attention) maintains high performance even at 60% filtration, significantly outperforming perplexity-based (PPL) methods.

The robustness of the method is particularly evident at high compression levels. While perplexity-based methods (PPL) show sharp performance declines as more text is filtered, AttnComp maintains over 90% of the original model's performance even when 60% of the context is removed. Furthermore, AttnComp is more efficient; by using a single pass through a helper model instead of iterative pruning, it reduces latency (e.g., 1.59s vs 1.72s for LongLLMLingua) while achieving higher compression.

Conclusion and Future Work

AttnComp shifts the paradigm of prompt compression from external statistical proxies to internal model mechanisms. By treating tokens as social units and using attention as a compass, it provides a more accurate and efficient way to handle long contexts.

Future research will focus on optimizing the graph-based clustering step, which can become a bottleneck for extremely long prompts, and exploring the impact of different helper models (e.g., Mistral vs. Llama) on compression quality. While some tasks like Qasper show slight sensitivity to pruning, the overall gains in efficiency and accuracy make AttnComp a powerful tool for scaling LLM applications.

Summary updated

BibTeX
@inproceedings{zhao2025leveraging,
  author = {Yunlong Zhao and Haoran Wu and Bo Xu},
  title = {{Leveraging Attention to Effectively Compress Prompts for Long-Context LLMs}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34800},
}
Figure