Tucker Attention: A generalization of approximate attention mechanisms

Timon Klein, Jonas Kusch, Sebastian Sager, Stefan Schnake, Steffen Schotthöfer

The Memory Bottleneck in Transformers

Multi-Head Attention (MHA) is the computational engine driving modern Transformers, but it comes at a steep memory cost. The massive weight matrices required for queries, keys, and values lead to inflated training costs—due to the storage of optimizer states—and create the infamous "KV-cache" bottleneck during inference.

To mitigate these issues, researchers have developed low-rank alternatives such as Grouped-Query Attention (GQA) and Multi-Head Latent Attention (MLA). However, these methods are largely heuristic. Crucially, they fail to fully exploit redundancies across the "head" dimension, ignoring the reality that different attention heads often learn highly correlated representations.

The Tucker Attention Framework

To address these inefficiencies, researchers propose Tucker Attention, a principled approach that shifts the perspective on attention weights. Instead of treating Query, Key, and Value matrices individually, Tucker Attention groups them into two 3rd-order tensors:

  1. Pre-softmax Tensor ($\mathcal{W}$): Captures the interaction between Queries and Keys.
  2. Post-softmax Tensor ($\tilde{\mathcal{W}}$): Captures the interaction between Values and Output projections.

By applying Tucker Decomposition—a classical linear algebra technique for tensor compression—these objects are factorized into a small "core" tensor and three basis matrices corresponding to the heads, the query/output dimension, and the key/value dimension.

This mathematical formalization proves that MHA, Multi-Query Attention (MQA), GQA, and MLA are not disparate architectures, but rather special cases of Tucker Attention with specific rank configurations.

Key Innovations and Insights

  • Compressibility of the "Head" Dimension: The study reveals that the number of "basis heads" can be drastically compressed. A model with 32 physical heads can often be represented by a much smaller set of basis vectors without a loss in accuracy.
  • Independent Rank Selection: Unlike MLA, which ties certain dimensions together, Tucker Attention allows researchers to independently tune the compression ranks for heads, queries, and keys, offering finer architectural control.
  • Latent RoPE: Tucker Attention introduces a method to apply Rotary Position Embeddings (RoPE) directly within the compressed latent space. This simplifies the complex "decoupled RoPE" seen in models like DeepSeek-V2, enabling full weight fusion at inference time for a simpler, faster model.
  • Hardware Compatibility: Despite the underlying tensor mathematics, Tucker Attention remains fully compatible with standard high-speed Flash-Attention kernels.

Empirical Results

The researchers evaluated Tucker Attention across a variety of vision (ImageNet, CIFAR) and language (OpenWebText, TinyShakespeare) tasks, utilizing architectures ranging from Vision Transformers (ViTs) to 1B-parameter LLaMA-3 style models.

  • Massive Parameter Savings: In ViT evaluations, Tucker Attention achieved accuracy comparable to GQA and MLA while utilizing an order of magnitude (10x) fewer parameters in the attention layers.
  • LLaMA-3 Efficiency: When training a 1B-parameter LLaMA-3 model from scratch, the method required only 10–20% of the attention parameters of the GQA baseline, incurring only a negligible 1–3% increase in validation loss.
  • Speed and Memory Reductions: The reduction in parameters and KV-cache size translated to a 20% decrease in iteration wall time compared to standard MHA, alongside a 4–8x reduction in the KV-cache footprint—a critical improvement for serving long-context LLMs on memory-constrained GPUs.

Limitations and Future Directions

While highly effective, Tucker Attention has a few limitations. The current implementation relies on standard Flash-Attention by formatting the Tucker structure to mimic Multi-Query Attention; developing a custom kernel designed specifically for the Tucker "core" could unlock even greater efficiencies. Furthermore, the method introduces three distinct rank hyperparameters ($r_1, r_2, r_3$), which requires additional tuning during model design. Finally, while results at the 1B-parameter scale are highly promising, the scaling dynamics of these low-rank approximations on frontier models (70B+ parameters) remain an open area for future research.

Summary updated

Abstract

The pursuit of reducing the memory footprint of the self-attention mechanism in multi-headed self attention (MHA) spawned a rich portfolio of methods, e.g., group-query attention (GQA) and multi-head latent attention (MLA). The methods leverage specialized low-rank factorizations across embedding dimensions or attention heads. From the point of view of classical low-rank approximation, these methods are unconventional and raise questions of which objects they really approximate and how to interpret the low-rank behavior of the resulting representations. To answer these questions, this work proposes a generalized view on the weight objects in the self-attention layer and a factorization strategy, which allows us to construct a parameter efficient scheme, called Tucker Attention. Tucker Attention requires an order of magnitude fewer parameters for comparable validation metrics, compared to GQA and MLA, as evaluated in LLM and ViT test cases. Additionally, Tucker Attention~encompasses GQA, MLA, MHA as special cases and is fully compatible with flash-attention and rotary position embeddings (RoPE). This generalization strategy yields insights of the actual ranks achieved by MHA, GQA, and MLA, and further enables simplifications for MLA.
BibTeX
@inproceedings{klein2026tucker,
  author = {Timon Klein and Jonas Kusch and Sebastian Sager and Stefan Schnake and Steffen Schotthöfer},
  title = {{Tucker Attention: A generalization of approximate attention mechanisms}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=ErcPPRZaiq},
}
Figure