In the standard Transformer architecture, every attention head maintains its own unique set of Queries, Keys, and Values. While this allows for rich, multi-faceted representations, it creates a massive "Memory Wall" during inference. As models generate text token-by-token, they must store the Key and Value (KV) vectors for every previous token in a "KV cache." For long sequences, this cache becomes so large that it exceeds GPU memory, forcing slow data transfers from main memory (DRAM).
The research direction explored here focuses on asymmetric attention architectures—specifically, reducing the number of Key and Value heads relative to Query heads to slash memory overhead without sacrificing model quality.
The Spectrum of Sharing: MQA and GQA
The most aggressive form of this idea is Multi-Query Attention (MQA). As Aakanksha Chowdhery and the PaLM team demonstrated, you can share a single set of Key and Value heads across all Query heads in a layer. This reduces the KV cache size by a factor equal to the number of heads (often 32x or more). Jia Li showed in StarCoder that this architectural shift allows for significantly larger batch sizes and faster processing, as the GPU spends less time moving data and more time computing.
However, MQA can sometimes lead to a slight dip in accuracy because the model has fewer "slots" to store different types of information. To find a middle ground, Team GLM and others adopted Grouped Query Attention (GQA) in models like ChatGLM. In GQA, Query heads are partitioned into groups, and each group shares one KV head. This provides a tunable balance: you get much of the memory savings of MQA with the representational power of standard Multi-Head Attention.
Why It Works: Data Movement Complexity
To understand why reducing heads improves speed, we must look at Data Movement Complexity (DMC). As Chen Ding 0001 argues in DMC4ML, the bottleneck in modern AI is often not the number of math operations (FLOPs), but the cost of moving data through the memory hierarchy.
In standard attention, the GPU must load a unique KV pair for every Query head from the slow DRAM into the fast on-chip SRAM. By sharing KV heads, GQA and MQA allow the GPU to load the KV data once and reuse it for multiple Queries. This transforms a memory-bound operation (waiting for data) into a compute-bound operation (calculating attention scores), which GPUs are natively designed to handle efficiently.
Advanced Compression: Latent Attention and Layer Condensation
Recent work has pushed the boundaries of sharing beyond just heads. Multi-Head Latent Attention (MLA), discussed by Ahmet Caner Yüzügüler in the context of TyphoonMLA, uses low-rank projections to compress the KV information into a latent vector. This allows the model to "absorb" the KV cache into a much smaller footprint while still allowing individual heads to reconstruct the specific information they need.
Other researchers, such as Haoyi Wu and Kewei Tu, have proposed Layer-Condensed KV Caching (LCKV). Instead of every layer having its own KV cache, LCKV allows multiple layers to attend to a shared KV representation from a single "anchor" layer. This vertical sharing across the depth of the network can lead to up to a 26x increase in throughput.
Amortizing the Cost of Shared Prefixes
A specific application of this sharing logic occurs when multiple requests share a common "system prompt" or "prefix." Lei Zhu introduced RelayAttention, which reformulates the attention math so that the shared prefix is read from memory exactly once per batch. Similarly, Bifurcated Attention splits the attention operation into two parallel paths—one for the shared prompt and one for the unique generated tokens—ensuring that redundant computations are eliminated.
Intuition Summary
The core insight of this research direction is that redundancy is an opportunity.
- Horizontal Redundancy: Multiple attention heads often look at the same patterns; sharing KV heads (MQA/GQA) exploits this.
- Vertical Redundancy: Different layers often process the same context; shared-layer caching (LCKV) exploits this.
- Batch Redundancy: Different users often start with the same prompt; shared-prefix kernels (RelayAttention) exploit this.
By strategically breaking the 1:1:1 ratio of Query:Key:Value heads, researchers are moving toward a more "asymmetric" Transformer that prioritizes the expensive memory bandwidth of Keys and Values while maintaining the expressive power of multiple Query heads.