In standard self-attention, the model computes relationships between tokens by multiplying a Query matrix $Q$ and a Key matrix $K$, resulting in an attention matrix $A = \text{softmax}(QK^T)$. Mathematically, if the head dimension is $d$ and the sequence length is $N$, the product $QK^T$ is the multiplication of an $N \times d$ matrix by a $d \times N$ matrix. By the properties of matrix rank, the rank of this product is strictly bounded by the head dimension: $\text{rank}(QK^T) \le d$.
Because $d$ is typically much smaller than $N$ (e.g., $d = 64$ while $N = 2048$), the attention matrix is fundamentally trapped in a low-rank bottleneck. While the softmax operation introduces non-linearity, the underlying representational capacity remains constrained. This bottleneck prevents individual attention heads from capturing highly complex, multi-faceted, or non-redundant relationships across long sequences. The dilemma becomes even more acute in linear attention variants, which omit the softmax operation entirely, capping the expressivity of the model to the feature dimension.
To resolve this, a prominent research direction focuses on attention rank expansion—architectural modifications that allow the mathematical rank of the attention matrix or internal state to scale beyond the physical constraints of a single head's dimension.
Additive Rank Expansion via Feature Decoupling
One of the most intuitive ways to bypass the low-rank bottleneck leverages a fundamental property of linear algebra: the rank of a sum of matrices can be equal to the sum of their individual ranks ($\text{rank}(A + B) \le \text{rank}(A) + \text{rank}(B)$). Rather than projecting all information into a single low-dimensional vector before computing attention, models can compute independent attention matrices from decoupled information sources and fuse them additively.
Early Integration (Low Rank):
[Tokens + Positions] ---> Q, K ---> Attention Matrix (Rank <= d)
Decoupled Integration (Rank Expanded):
Tokens ---> Q_t, K_t ---> Attention_t \
+ ---> Total Attention Matrix (Rank <= d_t + d_p)
Positions ---> Q_p, K_p ---> Attention_p /
For instance, in sequential recommendation systems, models traditionally compress item IDs and side information (like brands or categories) into a single embedding before the attention layer. Yueqi XIE et al. address this in Decoupled Side Information Fusion for Sequential Recommendation by decoupling these features during the attention calculation. By summing independent attention matrices derived from distinct feature sets, the effective rank of the final representation expands to the sum of the individual feature ranks, capturing intricate user behavior patterns that would otherwise be compressed away.
A similar bottleneck occurs with positional encodings. When position embeddings are simply added to token embeddings at the input layer, the total rank remains capped by the head dimension. Pu-Chin Chen et al. demonstrate in A Simple and Effective Positional Encoding for Transformers that injecting positional information directly at the attention head level overcomes this bottleneck. This per-head decoupling increases the total representational rank to the sum of the token and position dimensions, allowing the model to capture more intricate structural dependencies.
This additive principle is also highly effective for parameter-efficient fine-tuning. In APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models, Qifan Wang et al. show that unconstraining input-level soft prompts and inserting independent learnable parameters directly into the Query, Key, and Value matrices provides the attention mechanism with a much higher rank and greater representational capacity for downstream tasks.
High-Rank Expressivity via Structured Matrices and Mixtures
Another strategy replaces the standard deterministic point-vector projections with richer mathematical structures—such as probability distributions or tensor decompositions—to achieve high-rank expressivity without incurring quadratic computational costs.
Probabilistic Key Mixtures
Instead of mapping a token to a single deterministic key vector, we can model keys as a mixture of distributions. Dung D. Le et al. introduce this concept in Improving Transformers with Probabilistic Attention Keys by replacing the single key in each attention head with a Mixture of Gaussian Keys.
Because a query can match multiple sub-components of a key distribution simultaneously, the mathematical rank of the resulting attention matrix increases significantly. This allows a single head to attend to multiple diverse positions at once, mitigating head redundancy and allowing models with fewer heads to match the expressivity of much larger baselines.
Tensor Decompositions and Structured Operators
In domains like Spiking Neural Networks (SNNs), spatial-temporal attention maps are often mathematically equivalent to rank-1 tensor decompositions, severely limiting their flexibility. Haoyu Deng et al. break this limitation in Tensor Decomposition Based Attention Module for Spiking Neural Networks by generalizing these mechanisms to higher, adjustable ranks, which prevents overfitting and improves feature representation.
For standard Transformers, Yilun Kuang et al. target the low-rank bilinear transformation of the scoring function itself in Customizing the Inductive Biases of Softmax Attention using Structured Matrices. By utilizing Block Tensor-Train and Multi-Level Low Rank decompositions, the authors achieve full-rank expressivity with sub-quadratic parameter and operation counts. This allows the model to capture high-dimensional dependencies that are typically lost during standard low-dimensional projections.
Restoring Rank in Linear Attention and Recurrent States
Linear attention models drop the softmax operation to achieve $O(N)$ time and memory complexity, but this exacerbates the low-rank dilemma because the attention matrix rank is strictly bounded by the head dimension $d$. Several recent works focus on restoring this lost expressivity.
Standard Linear Attention:
Rank is strictly capped: Rank <= d
Rank-Augmented / Multi-Head Linear Attention:
Rank grows additively with blocks/partitions: Rank <= Block_1 + Block_2 + ...
+ Block_n
Token-Level Partitioning and Mixing
To close this expressivity gap, Kewei Zhang et al. propose MHLA: Restoring Expressivity of Linear Attention via Token-Level Multi-Head. By partitioning tokens into independent blocks and applying a learnable mixing mechanism, the rank of the attention matrix is allowed to grow additively with the number of blocks rather than being capped by the head dimension. This mathematical expansion restores the expressive power of quadratic softmax attention while maintaining linear complexity.
Similarly, Qihang Fan et al. introduce Rank-Augmented Linear Attention (RALA) in Breaking the Low-Rank Dilemma of Linear Attention, leveraging the property that the sum of multiple low-rank matrices forms a high-rank representation to restore feature diversity in vision tasks.
High-Rank State and Decay Dynamics
For recurrent and linear attention architectures that compress history into a hidden state matrix, the bottleneck often lies in how that state is updated over time. Traditional models use simple rank-1 updates to modify their internal memory.
Jiefu Zhang et al. address this in Householder-Diagonalized Linear Attention (HDLA): Utilizing Enhanced Decay Mechanism for Efficient Sequence Modeling. By utilizing generalized Householder matrices to diagonalize the decay matrix, the authors transition the model from standard rank-1 updates to a more sophisticated diagonal-plus-rank-2 decay structure. This increased rank allows the model to perform higher-fidelity memory retrieval and manage complex information over massive sequences.
When dealing with pre-trained models, Xingyu Shen et al. demonstrate in StateX: Enhancing RNN Recall via Post-training State Expansion that consolidating multiple small attention heads into a single large head allows the model to utilize larger, non-diagonal regions of the state matrix. This post-training state expansion dramatically improves long-context recall without requiring the model to be trained from scratch.
Summary of Rank Expansion Paradigms
| Paradigm | Core Mechanism | Key Benefit | Representative Work |
|---|---|---|---|
| Feature Decoupling | Computes separate attention matrices for distinct features (e.g., IDs, positions) and sums them. | Rank scales additively to the sum of feature dimensions. | Xie et al., Chen et al. |
| Probabilistic Mixtures | Replaces deterministic key vectors with a mixture of continuous distributions. | Single heads can map queries to multiple semantic regions simultaneously. | Le et al. |
| Structured Factorization | Uses Tensor-Train or high-order decompositions within the scoring function. | Achieves full-rank expressivity with sub-quadratic parameter scaling. | Kuang et al., Deng et al. |
| Block-Level Mixing | Partitions tokens into blocks and mixes them in linear attention. | Bypasses the strict head-dimension cap in linear attention models. | Zhang et al. (MHLA) |
| High-Rank Decay | Upgrades recurrent state updates from rank-1 to diagonal-plus-rank-2. | Enhances compressed memory fidelity for long-context retrieval. | Zhang et al. (HDLA) |
Key Foundational Work
Improving Transformers with Probabilistic Attention Keys (2022) — contributed 6 concept elements to this direction.
Decoupled Side Information Fusion for Sequential Recommendation (2022) — contributed 2 concept elements to this direction.
A Simple and Effective Positional Encoding for Transformers (2021) — contributed 2 concept elements to this direction.