Query Guided Context Compression

26 concepts16 papers3 proposals

As Large Language Models (LLMs) are increasingly deployed to reason over massive context windows, they encounter a severe computational bottleneck. Processing long documents incurs a heavy tax due to the quadratic complexity of attention mechanisms and the linear growth of Key-Value (KV) caches. A natural remedy is context compression: reducing the length of the input text before feeding it to the primary model.

However, conventional compression frameworks often operate in a query-agnostic (or task-agnostic) manner. They summarize or prune text based on general semantic importance or perplexity metrics before the user's specific question is known. This approach introduces a fundamental flaw: a detail that appears entirely redundant in a general context may be the exact "needle" required to answer a specific downstream query.

The Rate-Distortion Bottleneck

To understand why query-agnostic compression struggles, it is helpful to look at the problem through an information-theoretic lens. As formalized by Alliot Nagle et al. in a rate-distortion framework, a compressor that operates without knowledge of the downstream task faces a strict optimization barrier. Because it cannot predict which specific facts will be requested, it is forced to preserve all potentially relevant information to minimize downstream distortion. This constraint severely limits the achievable compression ratio.

When general-purpose compression methods attempt to aggressively shrink prompts, they inevitably treat all parts of a document with equal weight or rely on generic importance heuristics. As noted by Weizhi Fei et al. and Zhuoshi Pan et al., this task-blind reduction discards niche or localized information. Consequently, the model suffers from "information neglect" or the accidental deletion of critical facts needed for accurate reasoning (Jiwei Tang et al.).

Query-guided context compression shifts this rate-distortion frontier. By injecting the user's explicit question or instruction directly into the compression pipeline, the system can evaluate the utility of each token, clause, or activation relative to that specific objective. Information that is highly descriptive but irrelevant to the prompt can be safely and aggressively discarded, unlocking massive reduction ratios without degrading task accuracy.

Query-Agnostic:  [Long Document] ---> [Generic Compressor] ---> [Static Summary] ---> LLM + Query
                                                                        |
                                                          (Niche details are lost)

Query-Guided:    [Long Document] + [Query] ---> [Dynamic Compressor] ---> [Task-Specific Tokens] ---> LLM
                                                                                |
                                                                  (Only relevant info kept)

Mechanisms of Query-Aware Reduction

Modern research implements query guidance at different stages of the LLM inference pipeline, ranging from hard token filtering to dynamic memory management.

1. Hard Token and Attention Filtering

Instead of treating sentences as indivisible units, early foundational work by Stephen Wan et al. and Lu Wang et al. demonstrated that segmenting text into fine-grained clauses and filtering them based on query relevance prevents the accidental destruction of critical details.

In the context of modern LLMs, this is achieved by using lightweight "selector" models that compute attention alignment between the document and the query. For example, the QUITO framework introduced by Wenshan Wang et al. leverages a small, fraction-of-a-billion parameter model to filter out query-irrelevant tokens via attention scores. Similarly, Xuancheng Li et al. utilize an adaptive task-aware controller to dynamically adjust the compression rate on the fly, ensuring that only segments with high semantic relevance to the query are passed to the downstream model.

2. Soft Prompt and Recurrent Embedding Compression

Rather than making hard binary choices to keep or delete tokens, another approach compresses long contexts into dense, continuous vector representations—often called "soft prompts."

To prevent information loss during this process, the user's query must condition the embedding generation. In the LCIRC framework proposed by Sumin An et al., long inputs are recurrently compressed into compact representations where the query is injected via gated cross-attention. This ensures that the resulting compact activations prioritize task-relevant features over generic document properties.

For Retrieval-Augmented Generation (RAG) pipelines, Maxime Louis et al. developed OSCAR, which dynamically transforms retrieved documents into a small set of dense embeddings tailored on-the-fly to the user's query. By conditioning the compression directly on the question, these methods can achieve up to a 15x reduction in context size while maintaining or even improving downstream accuracy (Zhiwei Cao et al.).

3. Instruction-Aware KV Cache Eviction

Compression can also occur dynamically within the LLM's internal memory during generation. Standard KV cache eviction strategies typically rely on local fluency or recency heuristics (e.g., keeping the first and last few tokens). However, the information required to satisfy a specific instruction is often buried deep within the middle of a sequence.

To solve this, Yu Bai et al. developed CItruS, a method that uses the user's explicit instruction to guide state eviction, preventing the model from purging critical historical context. Similarly, the Finch framework by Giulio Corallo et al. uses the prompt to dynamically select and retain only the most relevant KV pairs in memory. This instruction-driven pruning slashes the KV cache footprint by up to 93x without requiring model retraining.

The Operational Trade-off

While query-guided compression offers superior efficiency and fidelity, it introduces a fundamental operational trade-off: latency vs. reusability.

Static, query-agnostic compression methods—such as the Activation Beacon approach by Peitian Zhang et al.—allow a document to be compressed once and stored in a cache. This compressed representation can then be reused across thousands of different user queries, making it highly cost-effective for static document databases.

Query-guided compression breaks this reusability. Because the compressed representation is strictly dependent on the specific question asked, the compression step must be performed online for every new query. As highlighted by Barys Liskavets et al., in open-ended scenarios like general document indexing or code analysis where an explicit question is unavailable at ingestion time, pure query-guided systems cannot be deployed without modification.

The field continues to navigate this frontier, balancing the theoretical limits of task-blind information reduction against the high-fidelity performance of query-driven prioritization.

Key Foundational Work

A Sentence Compression Based Framework to Query-Focused Multi-Document Summarization (2013) — contributed 3 concept elements to this direction.
Experimenting with Clause Segmentation for Text Summarization (2008) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_query_agnostic_compression_bec5fafa,
  title = {query-agnostic compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/query-agnostic-compression/rdr_bec5fafa47804e0a9f22c0296ed226b3},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_query_agnostic_compression_bec5fafa_v1,
  title = {query-agnostic compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/query-agnostic-compression/rdr_bec5fafa47804e0a9f22c0296ed226b3/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake