Semantic Context Compression for Large Language Models

31 concepts26 papers1 proposal

Large Language Models (LLMs) are fundamentally constrained by their context windows. Even as maximum token limits expand, the computational cost of self-attention scales quadratically ($O(N^2)$) with input length, making the processing of long documents prohibitively expensive. Furthermore, packing thousands of raw tokens into a prompt introduces significant noise, often causing models to miss critical information buried deep within the text—a phenomenon known as the "lost in the middle" problem.

To bypass these architectural limitations, a powerful paradigm has emerged: semantic context compression. Instead of feeding raw, redundant text directly into a downstream model or retriever, systems can compress long contexts into highly dense, informative representations. This approach filters out linguistic fluff while preserving the core semantic signals, allowing models to effectively process vastly larger volumes of information.

Textual Summarization as a Filter

The most intuitive way to compress context is to pass raw text through an initial summarization or extraction pipeline. Natural language is highly redundant; much of a document's token count is consumed by syntax, formatting, and non-essential elaboration. By using a smaller, specialized model to strip away this redundancy, systems can fit long-range information into tight token budgets.

This is often achieved via an "extract-then-abstract" approach. For instance, Bajaj et al. demonstrate that an initial extractive stage based on perplexity can filter long documents down to their most salient sentences before passing them to an abstractive summarizer, effectively overcoming the input limits of standard transformers. Similarly, Jin et al. utilize extractive summarization to condense multiple retrieved documents into a single, cohesive context window.

In highly technical domains like legal case law or clinical trials, simple truncation destroys vital evidence. To solve this, Askari et al., Abbas Akkasi et al., and Spandan Das et al. employ generative summarization modules to distill dense, long-form reports into concise representations before downstream inference. This ensures that the global semantic coherence of the source text is maintained.

[Raw Document: 10,000 Tokens] 
       │
       ▼ (Secondary Model / Extractive Filter)
[Compressed Summary: 500 Tokens] 
       │
       ▼ (Primary LLM / Downstream Inference)
[High-Quality Output]

This text-to-text compression can also be applied modularly or hierarchically:

  • Topic-Specific Chunking: Fei et al. show that summarizing independent, topic-specific text clusters using a smaller model can extend an LLM's effective context window by 6 to 8 times without requiring any parameter updates or fine-tuning.
  • Task-Specific Distillation: Cui et al. compress long-range history to improve document-level machine translation, while Xukai Liu et al. and Dong et al. compress long entity descriptions and document sections to prevent information loss in dense retrievers.
  • Asynchronous Offloading: Zhang et al. use a large cloud model to compress retrieved documents into concise summaries, drastically reducing the context window and latency requirements for smaller, real-time client models.

Structured Semantic Re-framing

Compression does not have to yield standard paragraphs. Text can be restructured into alternative formats that maximize information density and align better with a model's reasoning capabilities.

One approach is to transform unstructured text into structured Question-Answer (QA) pairs. Trisha Das et al. demonstrate that converting long, complex clinical protocols into a collection of QA pairs acts as an exceptionally effective semantic compression technique for information retrieval. This structured format strips away irrelevant text while explicitly mapping out the core factual relationships, preventing the information loss that typically occurs during standard truncation.

Alternatively, multi-document contexts can be re-framed as an answer sentence selection problem. Dernoncourt et al. bypass transformer length limits by decomposing long documents into individual sentences, scoring them with a binary classifier based on query relevance, and concatenating only the top-ranked sentences. Similarly, Haonan Li et al. use a selector-summarizer mechanism to distill long-form user reviews into representative sentences. For highly structured data objects, Dihan et al. show that stripping out redundant metadata and converting the objects into compact, human-readable formats optimizes token consumption and boosts reasoning efficiency.

Compressing Context into Dense Vectors and Tokens

The ultimate bottleneck of text-to-text compression is that natural language characters still require tokenization and processing through the model's embedding layers. To achieve maximum efficiency, researchers have looked toward compressing text directly into the model's internal representation space—turning thousands of words into a handful of "summary vectors" or "soft prompts."

Chevalier et al. pioneered this by adapting language models to compress long text segments into a small set of summary vectors. These vectors act as virtual tokens or soft prompts that can be appended to subsequent inputs, allowing the model to maintain performance over tens of thousands of tokens. Building on this concept, Tan et al. introduced LLoCO, a framework that combines dense summary embeddings with parameter-efficient fine-tuning (LoRA). By shifting the computational burden of long contexts from real-time attention to offline pre-computation, they enabled models with a native 4k context window to effectively evaluate up to 128k tokens.

[Raw Text Segment] ──► [Smaller Encoder Model] ──► [Single Semantic Token / Vector]
                                                               │
                                                               ▼
                                                  [Fed directly to Large LLM]

In its most extreme form, an entire document can be compressed into a single token or word:

  • Token-Level Encoding: Bo Chen et al. utilize a smaller, specialized language model to encode long-form text into a single representative token, allowing a larger model to process high-level semantic information with up to a 10× speedup in inference.
  • Constrained Vocabulary Prompting: Zhuang et al. prompt an LLM to represent a text using a single word. This constraint forces the model to compress the document's entire semantic essence into its final hidden state and output distribution, generating high-performance retrieval embeddings without expensive contrastive training.
  • Reconstructive Bottlenecks: Tanaka et al. employ a self-supervised task that forces a model to reconstruct the full textual content of a document from a single compressed summary vector, ensuring that the bottleneck token captures high-fidelity semantic information.

Why the Semantic Bottleneck Works

Semantic context compression works because it exploits an information-theoretic bottleneck. Raw natural language is incredibly sparse; the actual meaning or state-change intended by a paragraph can often be expressed in a fraction of the tokens.

When an LLM processes a massive, uncompressed prompt, its attention mechanism must distribute weights across a vast sea of tokens, many of which are purely structural or conversational. This dilutes the attention score of critical tokens.

By forcing the context through a semantic bottleneck—whether it is a text summary, a structured QA map, or a dense vector—the system acts as a lossy compression algorithm that is highly selective. It discards syntactic noise and retains only the core relational and factual signals. As a result, downstream models receive an input that is already optimized for reasoning, effectively bypassing the quadratic attention overhead and eliminating the noise that degrades long-context performance.

Key Foundational Work

UCSD-Adobe at MEDIQA 2021: Transfer Learning and Answer Sentence Selection for Medical Summarization (2021) — contributed 2 concept elements to this direction.
ForecastQA: A Question Answering Challenge for Event Forecasting with Temporal Text Data (2021) — contributed 1 concept elements to this direction.
Combining Lexical and Neural Retrieval with Longformer-based Summarization for Effective Case Law Retrieval (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_semantic_context_compression_9e5f1ca7,
  title = {semantic context compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-context-compression/rdr_9e5f1ca73fd14a0ab3428a79213e48d8},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_semantic_context_compression_9e5f1ca7_v1,
  title = {semantic context compression},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/semantic-context-compression/rdr_9e5f1ca73fd14a0ab3428a79213e48d8/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake