The standard Transformer attention mechanism treats every token in a sequence with equal structural granularity. To compute self-attention, an input sequence of length $N$ is projected into Queries ($Q$), Keys ($K$), and Values ($V$), each of length $N$. The core operation computes a pairwise similarity matrix $QK^T$, resulting in an $O(N^2)$ time and memory bottleneck. During autoregressive generation, this forces the Key-Value (KV) cache to grow linearly with sequence length, rapidly exhausting hardware memory limits.
However, this symmetric treatment of sequence tokens is fundamentally redundant. While a model needs high-resolution Queries to pinpoint exactly where to look and to maintain precise output resolution, the historical context it looks back at (represented by Keys and Values) contains massive local redundancy. Adjacent pixels in an image, consecutive frames in an audio clip, and neighboring words in a sentence share substantial mutual information.
By breaking the symmetry of attention—keeping Queries at full resolution while downsampling or compressing Keys and Values—we can reduce the attention matrix from $N \times N$ to $N \times M$ (where $M \ll N$). This shifts the computational and memory scaling from quadratic to linear relative to the sequence length, unlocking the ability to process massive contexts efficiently.
Spatial and Temporal Downsampling via Convolutions
The most direct way to compress Keys and Values is to apply strided operations during their initial projection phase. Instead of projecting each token independently using standard linear layers, we can use convolutional layers or pooling mechanisms to aggregate local token neighborhoods into a single, dense representation.
Standard Attention:
Queries (N) x Keys (N) ---> Attention Matrix (N x N)
Compressed Attention:
Queries (N) x Keys (M) ---> Attention Matrix (N x M) [where M = N / stride]
In early foundational work, Peter J. Liu et al. introduced memory-compressed attention for long-form text generation. By using strided convolutions to downsample the key and value sequences, the authors reduced the number of positions the query had to interact with, allowing the model to ingest significantly longer sequences while preserving global context.
This insight translates naturally across modalities:
- Computer Vision: In Vision Transformers (ViTs), spatial tokens correspond to image patches. Wu et al. introduced Convolutional Vision Transformers (CvT), which replace standard linear projections with strided convolutional projections for the key and value matrices. This effectively subsamples the sequence length within the attention block, lowering memory usage while maintaining high predictive accuracy. Similarly, Chang et al. and Kim et al. demonstrated that integrating convolutional layers as projections reduces computational complexity by a factor proportional to the square of the stride, allowing models to capture complex visual relationships without quadratic scaling costs.
- Audio and Time-Series Processing: Audio signals and multivariate time-series are notoriously long and redundant. Sant et al. leveraged convolutional key-value compression in speech translation to handle long audio sequences while preserving spatial and temporal resolution. For time-series classification, Qi Liu et al. sub-sampled the temporal resolution of keys and values before the attention calculation, maintaining a global receptive field at a fraction of the cost.
- Pooling Alternatives: Downsampling does not strictly require learned convolutions. Burchi et al. showed that average-pooling input sequences into discrete temporal patches before calculating attention weights achieves performance parity with more complex methods, confirming that high-resolution context is often unnecessary for robust feature extraction.
When downsampling the context, a key challenge is ensuring that the model does not lose the ability to locate fine-grained details. To address this, Jiacheng Liu et al. designed a controllable interpolation-extrapolation process for KV-downsampling. By keeping the queries at full resolution, the model retains its capacity to precisely locate and attend to information, which is vital for discriminative tasks like image classification and generation. This architectural principle can even be extended to the model's internal dimensions; for instance, Jaszczur et al. integrated convolutional layers into attention projections to compress hidden dimensions, reducing the model dimension complexity from $O(d_{model}^2)$ to $O(d_{model}^{1.5})$.
Iterative Chunking and Global Memory Sinks
While strided convolutions work exceptionally well for fixed-size inputs or moderately long sequences, they still scale linearly with the total sequence length. For ultra-long sequences—such as entire codebases or massive document collections—running a convolution over the entire sequence at once becomes a hardware bottleneck.
To bypass this, the input sequence can be segmented into fixed-size chunks and processed sequentially. Instead of maintaining a token-by-token history, the model compresses the historical context of entire chunks into a compact memory state.
Sequence: [--- Chunk 1 ---] [--- Chunk 2 ---] [--- Chunk 3 ---]
│ │ │
▼ ▼ ▼
Cache: [Summary State 1] [Summary State 2] [Summary State 3] ---> Linear/Constant Memory
Ma et al. proposed an iterative chunked attention strategy that divides the input sequence into segments processed sequentially. By concatenating compressed KV caches across chunks and applying specialized position encodings, they achieved linear memory scaling while preserving the model's ability to extrapolate beyond its training length.
Taking this compression to its logical extreme, Wei et al. developed the Recurrent Attention Transformer (RAT), which summarizes an entire chunk of sequence data into a single key-value pair. Storing only one summarized state per segment drastically reduces the memory footprint of the KV cache, yielding up to a $9\times$ speedup in generation throughput while maintaining global context.
For domain-specific tasks that require reasoning over massive inputs, this combination of chunking and compressed global caching is highly effective:
- Whole Slide Image Classification: In medical imaging, whole slide images contain millions of pixels, which are typically broken into thousands of patches. Bai et al. introduced a global memory sink that caches representations from previously processed image segments. This allows the model to integrate global context across extensive sequences without exceeding hardware memory limits during the processing of individual segments.
- Code Vulnerability Detection: Codebases can span hundreds of thousands of tokens. Lamprou et al. utilized a hybrid architecture that segments long sequences while maintaining a compressed global memory. This allows the model to accumulate context across an entire codebase, ensuring that information from early files remains accessible for reasoning in later parts of the sequence without incurring quadratic memory overhead.
Why It Works: The Asymmetry of Information Retrieval
To understand why key-value compression is so effective, it helps to think of the attention mechanism as a database query system. The Queries represent the current question the model is asking. The Keys represent the index tags of the database, and the Values represent the actual content.
If you compress the Queries, you reduce the resolution of the question, making it impossible to generate sharp, precise answers or locate exact positions. But if you compress the Keys and Values, you are simply summarizing the database. A high-resolution question ("What specific value was calculated in line 42?") can still successfully match against a summarized index ("This section contains the variable initialization block") if the summary retains the essential semantic boundaries.
By exploiting this asymmetry, KV compression eliminates the fluff of local token redundancy while preserving the global semantic landscape, allowing Transformers to scale to unprecedented sequence lengths with high efficiency.
Key Foundational Work
CvT: Introducing Convolutions to Vision Transformers (2021) — contributed 2 concept elements to this direction.
Generating Wikipedia by Summarizing Long Sequences (2018) — contributed 1 concept elements to this direction.
Sparse is Enough in Scaling Transformers (2021) — contributed 1 concept elements to this direction.