In a standard Transformer layer, the multi-head self-attention mechanism relies on three distinct linear projections. The input tensor is multiplied by three separate weight matrices to create Queries ($Q$), Keys ($K$), and Values ($V$):
$$Q = XW_q, \quad K = XW_k, \quad V = XW_v$$
As models scale up in hidden dimensions and number of heads, these projection matrices consume a massive portion of the parameter budget. The core insight of parameter-efficient attention is that these projection spaces are highly redundant. The features required to determine where to look ($Q$ and $K$) and what information to extract ($K$ and $V$) overlap significantly. By sharing, consolidating, or eliminating these linear projections, we can drastically reduce model size with minimal impact on representational capacity.
Symmetric Attention: Unifying Queries and Keys
In standard attention, the affinity matrix is computed via the dot product $Q K^T$. Because $Q$ and $K$ both map tokens from the same input space to compute mutual relationships, their projection matrices often learn highly correlated features.
In the Reformer architecture, Kitaev et al. demonstrated that sharing the weights for query and key projections ($W_q = W_k$) simplifies hashing-based routing and reduces the overall parameter count of the attention block without degrading accuracy.
Building on this concept of symmetric feature spaces, Ostendorff et al. introduced a pairwise dot-product attention mechanism that enforces strict feature sharing between Queries and Keys. By mapping tokens into a single shared latent space, their approach eliminates projection redundancy and allows BERT-style language models to reach peak performance in half the training steps.
From a structural perspective, Yoshihiro Yamada consolidated these distinct projections into a single unified operation in CAT: Circular-Convolutional Attention for Sub-Quadratic Transformers, proving that high-dimensional redundancy in standard multi-head attention can be streamlined via the Fast Fourier Transform without sacrificing global softmax weighting.
This redundancy is also evident during model adaptation. Yao et al. provided theoretical insights showing that tuning both Query and Key matrices during fine-tuning is mathematically redundant because they interact strictly as a product. By focusing parameter updates exclusively on the Query and Value matrices, the authors achieved a 33% reduction in trainable parameters while improving generalization and convergence speed.
Context Consolidation: Sharing Keys and Values
Another major axis of redundancy lies between Keys and Values. Both vectors represent the context tokens being attended to, meaning they share the same underlying token distributions.
To exploit this, Tan et al. introduced key-value weight sharing ($W_k = W_v$) in their Compact Object Relation Transformer (ACORT). This modification effectively eliminates half of the context-projection parameters within the multi-head attention module, facilitating efficient deployment on resource-constrained hardware with negligible impact on representational quality.
Similarly, Zhao et al. leveraged shared key-value projection weights in MobileDiffusion, reducing the model's total parameter count by approximately 5% to enable sub-second text-to-image generation on mobile devices. For low-level vision tasks, Zhang et al. equated keys and values to eliminate computational redundancy and improve inference speed in image super-resolution.
Standard Attention: [Input X] ---> Projections W_q, W_k, W_v ---> Q, K, V
Shared KV Attention: [Input X] ---> Projections W_q, W_kv -------> Q, KV
Shared QKV Attention: [Input X] ---> Projection W_shared -------> QKV
This principle scales effectively to multi-expert architectures. Zhang et al. developed the BAM (Branch-Attend-Mix) framework for Mixture of Experts (MoE) models. Instead of duplicating entire attention blocks for each expert, they shared the Key and Value matrices across all experts while keeping the Query matrices unique. This configuration minimizes the model's KV-cache footprint during inference while still allowing individual experts to learn specialized attention patterns through their independent query projections.
Asymmetric Allocation and Extreme Sharing
If projections are not perfectly shared, parameter efficiency can still be achieved by allocating capacity where it matters most, or by eliminating projections entirely.
- Value-Centric Allocation: Liu et al. observed in EfficientViT that value projections require a larger capacity to preserve essential information, whereas query and key projections are far more amenable to aggressive parameter reduction. Conversely, Hazarika et al. constrained updates to the value matrix by using the frozen key matrix as a mathematical basis, minimizing the number of trainable parameters by reusing existing architectural structures to define the update space.
- Total Projection Unification: In the most extreme case of sharing, Yi et al. proposed a Shared Weight Self-Attention (SWSA) mechanism. SWSA uses a single shared weight matrix ($W_q = W_k = W_v$) to project inputs into a common representation space for all three calculations, drastically reducing the parameter footprint by two-thirds without compromising the model's ability to capture global dependencies.
- Identity Mapping: Taking a structural approach, He et al. demonstrated that in architectures stripped of skip connections and normalization layers, attention value and projection matrices naturally converge toward identity mappings. By fixing these matrices as identity from the start, they cut the attention sub-block's parameter count by 50% while maintaining the model's ability to learn complex representations.
Structural Replacements for Dense Projections
Beyond weight sharing, a parallel strategy replaces dense linear projections entirely with lightweight structural alternatives.
For instance, Luo et al. substituted large weight matrices with small, window-based filters derived from Feedforward Sequential Memory Networks (FSMN), cutting total model parameters by 20% for speech recognition. To achieve real-time inference on standard CPU hardware, Mitchell et al. replaced standard dense layers within transformer blocks with lightweight 1D convolutions.
Finally, Ji et al. bypassed deep stacks of attention layers entirely, using a single layer governed by an iterative dynamic routing mechanism to mitigate the parameter explosion common in multi-modal reasoning tasks.
Key Foundational Work
Simplified Self-Attention for Transformer-based End-to-End Speech Recognition (2020) — contributed 2 concept elements to this direction.
A Time Delay Neural Network with Shared Weight Self-Attention for Small-Footprint Keyword Spotting (2019) — contributed 2 concept elements to this direction.
Transformer Dissection: An Unified Understanding for Transformer's Attention via the Lens of Kernel (2019) — contributed 2 concept elements to this direction.