The standard Transformer architecture treats every layer as an independent computational island. In a model with dozens of layers, each layer projects its own unique set of Queries ($Q$), Keys ($K$), and Values ($V$) to compute attention. During autoregressive generation, the resulting Key and Value vectors must be stored in a memory buffer—the KV cache—to avoid recomputing them for past tokens at every step.
This design introduces a severe memory bottleneck. The size of the KV cache scales linearly with the sequence length, batch size, number of heads, and crucially, the number of layers. For long-context generation, the KV cache quickly balloons to gigabytes, dwarfing the model weights themselves and making inference heavily memory-bandwidth bound.
However, a fundamental question arises: do different layers actually need completely unique views of the past? A growing body of research demonstrates that representations change smoothly across layers and that adjacent layers often exhibit highly redundant attention patterns. By exploiting this inter-layer redundancy, we can share Key-Value states or attention maps across layers, breaking the linear memory tax of deep Transformers without sacrificing performance.
The Intuition of Inter-Layer Redundancy
To understand why cross-layer sharing works, consider what happens to a token's representation as it travels up a deep Transformer network. Lower layers tend to focus on local syntactic or phonetic features, while upper layers integrate global semantic context. However, from one layer to the next, the hidden states change incrementally.
Empirically, if you look at the attention matrices of Layer $l$ and Layer $l+1$, they often look remarkably similar—they attend to the same sets of tokens with nearly identical weights. If two layers are looking at the same things and extracting similar features, computing and storing separate $K$ and $V$ projections for both layers is fundamentally redundant.
We can exploit this insight at three distinct levels of the attention mechanism:
- The Data Level: Sharing the raw Key and Value projections across layers.
- The Score Level: Sharing the computed post-softmax attention maps directly.
- The Representation Level: Decoupling and recycling the Value heads across the network.
Sharing Key-Value Projections Across Layers
The most direct way to shrink the KV cache is to force multiple layers to share the exact same Key and Value projections. Instead of allocating $L$ distinct caches for an $L$-layer model, the model is re-architected to reuse a single set of keys and values across a block of layers or even the entire network.
In an early exploration of this concept, Yan et al. introduced EL-Attention: Memory Efficient Lossless Attention for Generation, which shares a unified key and value representation across all heads and all layers of a decoder. This approach increases the arithmetic intensity of the attention operation and drastically reduces memory movement during incremental decoding.
To balance model expressivity with memory savings, Brandon et al. proposed Cross-Layer Attention (CLA) in Reducing Transformer Key-Value Cache Size with Cross-Layer Attention. Instead of a single global cache, CLA shares KV activations across adjacent pairs or small blocks of layers. When combined with Multi-Query Attention (MQA), this achieves a multi-fold reduction in cache size while remaining on the Pareto frontier for accuracy.
Taking this concept to its logical conclusion, Sun et al. designed You Only Cache Once: Decoder-Decoder Architectures for Language Models (YOCO). As shown below, YOCO splits the model into two distinct halves: a self-decoder that generates a single global KV cache, and a cross-decoder that efficiently reuses this single cache across all subsequent layers. This allows models to scale to million-token sequences on standard hardware.
Standard Transformer:
Layer 1: Hidden States ──► Compute K,V ──► Store in Cache 1
Layer 2: Hidden States ──► Compute K,V ──► Store in Cache 2
Layer 3: Hidden States ──► Compute K,V ──► Store in Cache 3
Cross-Layer Sharing (e.g., YOCO):
Layer 1: Hidden States ──► Compute K,V ──► Store in Global Cache
Layer 2: Hidden States ───────────────────► Reuse Global Cache
Layer 3: Hidden States ───────────────────► Reuse Global Cache
Rather than grouping layers statically, Zuhri et al. developed MLKV: Multi-Layer Key-Value Heads for Memory Efficient Transformer Decoding. MLKV allows multiple query heads across different layers to map to the same underlying KV heads, breaking the traditional architectural constraint that requires every layer to host at least one unique KV head. For existing, pre-trained models, Wang et al. demonstrated in CommonKV: Compressing KV Cache with Cross-layer Parameter Sharing that one can cache a lower-dimensional latent representation instead of the final projected vectors, exploiting the high similarity of hidden states across adjacent layers for training-free compression.
Reusing Attention Maps Directly
While sharing KV projections reduces storage, it still requires each layer to perform the $Q K^T$ matrix multiplication and the subsequent softmax operation to find out where to look. A second paradigm bypasses these operations entirely by sharing the final attention scores or weights across layers.
Xiao et al. first demonstrated this in Sharing Attention Weights for Fast Transformer, noting that adjacent decoder layers often focus on identical parts of a sentence. By reusing the attention weight matrices, the model avoids recalculating similar focus points, accelerating inference.
In Beyond KV Caching: Shared Attention for Efficient LLMs, Vargas extended this by sharing post-softmax attention scores across multiple layers. This not only shrinks the KV cache but also eliminates the computationally intensive softmax and dot-product operations in the receiving layers, significantly lowering total floating-point operations (FLOPs).
This redundancy is not unique to language. In speech processing, Shim et al. showed in Understanding the Role of Self Attention for Efficient Speech Recognition that self-attention exhibits distinct phonetic and linguistic localization blocks; computing the attention map once per block and sharing it yields massive training and inference savings. Similarly, for computer vision, Moitra et al. introduced TReX- Reusing Vision Transformer's Attention for Efficient Xbar-based Computing, which reuses attention maps across Vision Transformer (ViT) layers to eliminate redundant matrix multiplications on In-Memory Computing hardware.
Decoupling and Recycling Value Representations
A third facet of this research direction focuses specifically on the Value ($V$) vector. While Keys dictate where to look, Values represent what information is extracted. Researchers have found that the underlying semantic values can be decoupled from layer depth entirely.
In Value Residual Learning, Zhou et al. proposed utilizing a single, shared set of value embeddings across all layers of a network. By adding a shortcut for value vectors from the first layer to deeper ones, the network retains initial token-level information that is typically lost in deep architectures, cutting KV cache requirements by nearly half.
Building on this concept, Wu et al. developed SkipV1Former, detailed in Improving Model Representation and Reducing KV Cache via Skip Connections with First Value Heads. Their architecture computes the full set of value heads only in the initial layer, and then reuses a subset of these heads in all subsequent layers by concatenating them with a smaller set of fresh, layer-specific heads. This selective reuse bypasses the typical trade-off between memory efficiency and representation quality, actually lowering model perplexity.
Taking representation sharing a step further, the authors of Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation introduced SambaY, which uses an element-wise Gated Memory Unit (GMU) to share internal representations across layers. This gating mechanism effectively replaces full cross-attention mechanisms in deep architectures, drastically lowering input/output overhead during long-context reasoning tasks.
Enabling Advanced Inference Pipelines
Beyond direct memory savings, cross-layer cache sharing serves as an enabling mechanism for other advanced inference optimizations, such as speculative decoding and hierarchical sparse attention.
In speculative decoding, a smaller "draft" engine guesses tokens ahead, and a larger "target" model validates them. Traditionally, this requires deploying two separate models with separate caches. Liu et al. bypassed this in Speculative Decoding via Early-exiting for Faster LLM Inference with Thompson Sampling Control Mechanism by using the target model's own initial layers as the drafting engine, allowing the draft and target steps to share the same KV cache. Similarly, Li et al. developed Gumiho: A Hybrid Architecture to Prioritize Early Tokens in Speculative Decoding, which reuses KV caches across identical serial drafting heads to simplify state management and reduce memory overhead.
Speculative Decoding with Shared Cache:
[Input Tokens] ──► [Initial Layers / Draft Heads] ──► Generates Speculative Tokens
│
▼ (Shared KV Cache)
[Deeper Target Layers] ──► Validates Tokens Losslessly
Cross-layer sharing also optimizes sparse attention frameworks. In HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing, Cao et al. interleaved standard full-attention layers with multiple sparse attention layers. The full-attention layers act as "oracles" to identify important tokens and generate a global KV cache, which is then shared with and reused by the subsequent sparse layers, eliminating the need for every layer to maintain its own global context.
Summary
The traditional approach of computing unique attention states for every single layer in a Transformer introduces massive, redundant memory overhead. By recognizing that attention patterns and hidden states evolve gradually, cross-layer sharing techniques compress the KV cache at the data, score, or representation level. Whether through sharing raw projections, recycling post-softmax attention maps, or reusing value heads, this direction provides a path toward processing massive context windows on standard hardware.
Key Foundational Work
EL-Attention: Memory Efficient Lossless Attention for Generation (2021) — contributed 3 concept elements to this direction.
Sharing Attention Weights for Fast Transformer (2019) — contributed 1 concept elements to this direction.