When a language model generates text one token at a time, most of the cost is moving weights from GPU memory to the compute units, not the arithmetic itself. A compressed model is therefore only faster if it reads fewer bytes per token. Inside a transformer block, several weight matrices are multiplied by the same input vector. The query, key and value projections of every attention head all read the same hidden state, and the gate and up projections of the feed-forward layer read the same hidden state as each other. We propose to treat each such group as one multi-way array, a tensor, and factor it with a Tucker decomposition. A Tucker decomposition writes a tensor as a small "core" tensor multiplied by one basis matrix per dimension. Here it gives the whole group a single shared input basis, computed once per token, plus small per-matrix cores. The decomposition is weighted by the statistics of the real activations that flow into the group, so it preserves what the layer computes on actual data rather than the raw weight entries.
The core claim is that the shared-input structure and the activation weighting matter together and each fails alone. The structure makes the projection onto the input basis a single small dense matrix multiply shared by many matrices, which turns parameter savings into fewer bytes read and lower latency. The weighting prevents the basis from wasting capacity on directions the inputs never use. The proposal also targets a disagreement in the literature. Data-free tensor methods report strong results without looking at activations, while matrix methods report that ignoring activations is damaging. We test that disagreement directly at equal memory traffic.
Where this sits
Three tensor methods already factor attention weights jointly across heads. TensorLLM stacks the four attention projections into a fourth-order tensor with shared factors and per-head cores, mainly as a denoiser. LeSTD runs the same kind of shared-subspace decomposition and then prunes core entries without structure. TRAWL stacks related matrices for CP and Tucker denoising. It finds that stacking across all layers fails because layers differ too much, while stacking within a layer helps. All three are data-free and choose ranks by grid search, and these papers motivate the shared-basis idea.
The weighting comes from two other lines of work. TD-MoE supplies the method we reuse. It stacks mixture-of-experts feed-forward matrices, decorrelates inputs and outputs with activation covariances before Tucker (called multilinear whitening), and allocates the three ranks under a budget, but it never touches attention or dense models. The second line works on single matrices. ASVD scales each matrix by activation magnitudes before a matrix factorization (SVD), and AIR adds a gradient-based influence weighting. Both show that activation-aware factorization matters, but they handle each matrix separately. ASVD also reports that feed-forward layers resist low-rank compression, and our shared gate/up treatment is aimed at that weakness.
Our design differs from TensorLLM and LeSTD in three ways. We group matrices by the input they consume rather than by layer type. We weight the decomposition by input covariance. We judge results by bytes moved per decoded token. We differ from TD-MoE in applying whitened Tucker to attention heads and to gate/up pairs of dense models. 3BASiL is relevant in two ways. It supplies the optional block-level output-matching refinement. It is also a reminder that a compressed format only speeds things up if GPU kernels exist for its structure. This constrains us to dense small cores and rules out LeSTD-style unstructured sparsity.
Core Claim
For a group of $K$ matrices $W_k \in \mathbb{R}^{m_k \times d}$ sharing input activations $X \in \mathbb{R}^{d \times n}$, we compute $L$, the Cholesky factor of the input covariance $XX^\top/n$ (regularized). We then solve
$$\min_{U, \{V_k, G_k\}} \sum_{k=1}^{K} \big\| (W_k - V_k G_k U^\top) L \big\|_F^2 .$$
Here $U \in \mathbb{R}^{d \times r}$ is the shared input basis, $V_k$ is the per-matrix output basis, and $G_k$ is a small dense core. This is a Tucker problem on the whitened stacked tensor and can be solved by higher-order orthogonal iteration (alternating updates of each basis). The effective input basis $L^{-\top}$-corrected $U$ is folded into the stored factor, so whitening adds no inference cost, as in TD-MoE.
At decode time, $z = U^\top x$ is one $r \times d$ read, and each matrix then costs only $V_k G_k z$. The per-token bytes are roughly
$$B \approx b\,\Big(rd + \sum_k (r\,r_k + m_k r_k)\Big),$$
where $b$ is bytes per weight. Ranks $r$ and $r_k$ are allocated across layers by greedily trading a small calibration-loss increase per byte saved, under a total byte budget.
For grouped-query attention models, each key and value head serves several query heads. We still group them, because they read the same input. The output projection and the down projection read different inputs, so they get separate whitened decompositions.
Research Program
The work proceeds in three stages. First, we implement the whitened shared-input decomposition and the byte-budget rank allocator for attention input projections. Second, we extend it to gate/up pairs. Third, we write a fused decode kernel that computes $U^\top x$ once and then the batched small cores in one pass, and we measure tokens per second at batch size one and eight.
We use 256 WikiText-2 calibration samples, matching TD-MoE, and repeat with C4 calibration to check overfitting to the calibration domain. An optional stage adds block-level output matching in the style of 3BASiL to test how much a cheap refinement recovers. The deliverable is a perplexity-versus-measured-latency curve and a perplexity-versus-bytes curve. Parameter count alone is not reported as a result, because the file documents that low-rank products can fail to speed up decoding.
Baselines
The naive reference point is the uncompressed model, together with plain truncated SVD of each matrix at the same byte budget. Plain SVD is the floor any structured method must clear.
The strongest existing matrix method is AIR, run with its released recipe on the same calibration data, the same models, and the same byte budget instead of a parameter-retention target. The file reports that AIR improves perplexity over SVD-LLM by more than 18% at up to 60% parameter retention, with a 53% per-token latency reduction. Beating it means lower WikiText-2 and C4 perplexity at equal measured latency.
The strongest existing tensor method is LeSTD. The file reports WikiText-2 perplexity 42.70 for LeSTD versus 50.25 for SVD-LLM on OPT-30B at a 0.2 compression ratio. We will rerun it on OPT-6.7B and LLaMA-3-8B, and on OPT-30B if compute allows, so the cited number can be reproduced. LeSTD's speed will be measured with the best available sparse kernel. TensorLLM serves as a second data-free tensor baseline at matched bytes. No exact tabulated accuracy figures for TensorLLM were found on Lacuna, only a figure.
Two ablations isolate the key ingredient. The first is whitened per-matrix SVD (ASVD/SVD-LLM style), which has the weighting but no shared basis. The second is unwhitened shared Tucker, which is essentially LeSTD stage one with a dense core and has the shared basis but no weighting. If the full method does not beat both ablations at equal latency, the core claim fails. Every method gets the same rank-allocation search budget measured in GPU-hours.
Group-wise 4-bit quantization is a further reference. It is run both alone and stacked with each low-rank method, because the quantization alternative below may be stronger than any low-rank method on its own.
Why not just…?
Why not just use whitened per-matrix SVD? This is the most serious threat. If activation weighting accounts for all the benefit, then ASVD-style whitening per matrix should match the shared basis. The file's evidence that sharing helps comes from data-free settings (TensorLLM, TRAWL) and mixture-of-experts layers (TD-MoE beating per-expert SVD by more than 11% relative at 40% compression), not from whitened attention. The first ablation answers this directly. If whitened per-matrix SVD ties at equal latency, we drop the tensor claim and report the result as a negative finding.
Why not just quantize to 4 bits? Quantization cuts bytes per weight about fourfold with mature kernels, and at mild budgets it will likely beat any low-rank method. We do not argue otherwise. The relevant question is whether shared-input Tucker adds savings on top of 4-bit weights. DecoQuant and QDoTA suggest that decomposition and quantization can compose. We therefore report 4-bit alone versus 4-bit plus our method at equal total bytes. If the combination does not beat 4-bit alone, the method has no deployment value.
Why not just prune attention heads or channels? Structured pruning such as AdaPruner removes whole units and runs on dense kernels with no new code. It discards capacity that a shared basis could keep in compressed form. We include it at matched bytes on the zero-shot suite. No reported result on our exact models was found on Lacuna, so we will run it ourselves.
Why not just use LeSTD's data-free decomposition? It needs no calibration data, which is a real advantage. However, its sparse core needs unstructured sparse kernels, and AIR and ASVD show that ignoring activations hurts. The second ablation, compared against our full method and against LeSTD with its sparse core, will show whether whitening or core sparsity is worth more per byte.
Proposed Experiments
Models are LLaMA-3-8B as a dense grouped-query model and OPT-6.7B for comparability with LeSTD. Budgets are 80%, 60% and 40% of attention-plus-feed-forward bytes. We measure perplexity on WikiText-2, C4 and PTB, zero-shot accuracy on BoolQ, PIQA, HellaSwag, Winogrande, ARC-e/c and OBQA, MMLU, and GSM8K and MathQA as reasoning checks that are known to break under compression. Decode latency is measured on one A100 at contexts of 512 and 4,096.
Three outcomes are decisive. The first is whether the full method beats both ablations at equal latency. The second is whether gate/up sharing narrows the feed-forward gap that ASVD reports, judged by the perplexity cost of compressing feed-forward layers alone. The third is whether gains survive stacking with 4-bit quantization. We also report the cross-domain calibration check, WikiText-2 calibration evaluated on C4, because whitening can overfit to the calibration domain.
References
- TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs
- LeSTD: LLM Compression via Learning-based Sparse Tensor Decomposition
- TRAWL: Tensor Reduced and Approximated Weights for Large Language Models
- TD-MoE: Tensor Decomposition for MoE Models
- ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
- Activation- and Influence-Aware Ranks (AIR): Function-Preserving SVD Compression for LLMs
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
- Sample-aware Adaptive Structured Pruning for Large Language Models