Large Language Models (LLMs) face a fundamental scalability crisis: the computational cost of self-attention grows quadratically with the length of the input. While expanding context windows allows models to "see" more information, the resulting memory overhead and inference latency often make long-context processing impractical on standard hardware. As noted in Prompt Compression for Large Language Models: A Survey by Zongqian Li et al., the ballooning size of prompts requires new methods to "zip" instructions and contexts without sacrificing performance.
The emerging solution is soft context compression. Rather than pruning words or summarizing text into shorter natural language, these methods distill extensive information into a small set of dense, continuous "latent tokens" or "memory embeddings."
The Latent Bottleneck
The core intuition behind soft compression is that natural language is highly redundant. A Transformer does not necessarily need to attend to every discrete token in a 10,000-word document to understand its core logic or retrieve a specific fact. By forcing the context through a bottleneck, researchers can represent variable-length text as a fixed number of high-dimensional vectors.
Leo Feng et al. demonstrate this in Latent Bottlenecked Attentive Neural Processes, where context data is summarized into a fixed set of latent vectors. This architectural shift reduces memory usage by up to 11x, transforming the quadratic cost of attention into a linear one. Similarly, Alexis Chevalier et al. show in Adapting Language Models to Compress Contexts that specialized tokens can trigger the generation of "soft prompts"—compact vector representations that allow the model to attend to historical information without re-processing the raw text.
Virtual Tokens and Gisting
One prominent implementation of this bottleneck is the use of "virtual" or "gist" tokens. In Learning to Compress Prompts with Gist Tokens, Jesse Mu et al. train models to squeeze complex instructions into a tiny, cacheable prefix. These tokens act as a dense representation of the original context, enabling up to 26x compression.
This concept is extended to specific tasks like In-Context Learning (ICL) and Retrieval-Augmented Generation (RAG):
- In-Context Learning: Jun Gao et al. propose UniICL, which distills long demonstrations into "Memory Tokens" stored in a bank, allowing for 64-shot learning on consumer-grade hardware.
- Retrieval: Maxime Louis et al. introduce OSCAR, which uses a small compressor model to turn retrieved documents into dense soft embeddings on-the-fly, tailored to a user's specific query.
- Procedural Memory: Zijun Wu et al. show in TokMem that complex task instructions can be distilled into a single trainable embedding token, enabling models to execute thousands of distinct behaviors with constant-size context overhead.
Distilling Semantic Essence
Why does soft compression often outperform raw text? The authors of Soft Knowledge Prompt, Qunbo Wang et al., argue that transforming external information into a compact latent vector minimizes the impact of "linguistic noise" found in natural language. By mining pivotal information and projecting it into the model's embedding space, the system receives a cleaner, more effective signal than a raw text snippet.
This distillation process can be applied even to the model's internal states. In MELODI, Yinpeng Chen et al. demonstrate that Key-Value (KV) caches are highly compressible. By summarizing token activations into a smaller set of tokens, they capture the essential context needed for long-range dependencies while slashing the memory footprint by a factor of 8. For reasoning tasks, Zeliang Zhang et al. propose Progressive Thought Encoding, which encodes evicted KV cache tokens into lightweight adapters, translating discrete history into a latent representation stored within the model's parameters.
Efficient Architectures for Compression
The shift toward soft compression also invites architectural optimization. Edouardo Honig et al. found in Better Prompt Compression Without Multi-Layer Perceptrons that compressors do not need to be full-scale models; an "Attention-Only Compressor" with 67% fewer parameters can actually outperform heavier baselines.
Similarly, Jianxun Lian et al. developed PCC (Pretraining Context Compressor), a decoupled framework that uses fixed-length "memory slots" to represent variable-length text. By concentrating semantic information into these continuous slots, the authors achieved 16x compression with minimal performance loss, bypassing the token-level constraints that typically limit Transformer inference.
Key Foundational Work
Adapting Language Models to Compress Contexts (2023) — contributed 2 concept elements to this direction.
Learning to Compress Prompts with Gist Tokens (2023) — contributed 1 concept elements to this direction.
Latent Bottlenecked Attentive Neural Processes (2023) — contributed 1 concept elements to this direction.