Tensor-Train Rotations Around a 2:4 Sparse Core

GPUs from NVIDIA's Ampere generation onward can skip zeros in one specific pattern. In "2:4 sparsity", exactly two of every four consecutive weights are zero, and the hardware runs such a matrix multiply about twice as fast. The catch is that forcing a pretrained language model into this rigid pattern damages it badly. The best current fix, ARMOR, first rotates the weight matrix into a better basis using two cheap block-diagonal matrices, then prunes it to 2:4 in that basis. Block-diagonal rotations are cheap, but they cannot mix information across blocks. Their reach is limited by the block size, and the block size is limited by runtime cost.

We propose to replace those block-diagonal rotations with rotations written as a matrix product operator (MPO). An MPO is a chain of small tensors borrowed from the density matrix renormalization group (DMRG), the physicist's method for compressing quantum states. It can couple every input channel to every other at sub-quadratic cost. We also borrow DMRG's optimization machinery to fit these rotations to calibration data. The question is narrow and falsifiable: at a fixed wall-clock overhead on a real GPU, does a tensor-train rotation recover more of the 2:4 accuracy gap than a block-diagonal one?

Core Claim

We write a compressed linear layer as

$$\hat W = Q_{\text{out}} \,(S \odot M)\, Q_{\text{in}},$$

where $S \odot M$ is a 2:4-sparse core ($M$ is the binary 2:4 mask) and $Q_{\text{out}}, Q_{\text{in}}$ are orthogonal matrices. Each orthogonal matrix is stored as an MPO: the $N$ channel indices are factored as $N = d^n$, and the matrix becomes a chain of $n$ small tensors joined by "bond" indices of size $D$. The bond dimension $D$ limits how much the two sides of any cut in the chain can be correlated.

Applying such a rotation to an activation vector costs about $n N d D^2$ operations, against $N^2$ for the layer itself. For $N=4096$, $d=4$, $D=4$ that is roughly a tenth of the dense layer. This sits in the same budget range as ARMOR's 3–5% parameter overhead, and it is tunable through $D$. A block-diagonal matrix with $k$ blocks is itself an MPO whose bond dimension along the block index is $k$, but its blocks never talk to each other. The MPO family therefore strictly contains ARMOR's wrappers at comparable cost, and it adds cross-block mixing.

The second half of the claim is about optimization. DMRG keeps all tensors except one ("the orthogonality center") as isometries, meaning their columns are orthonormal. In that arrangement, the error of a local update equals the global error (Variational Ground State Optimization with Matrix Product States, which uses this canonical form to optimize infinite chains). We place the sparse core permanently at the orthogonality center and keep both wrapper chains isometric.

This placement resolves the tension the research file flags under its third gap. In plain DMRG, pruning MPS entries would break the orthogonality conditions and destabilize sweeping. Here only the center is pruned, and the wrappers stay exactly orthogonal by construction.

Research Program

Optimization. The layer-wise objective is the activation-weighted reconstruction error $\|W X - \hat W X\|_F^2$ on calibration activations $X$. Because $Q_{\text{out}}$ is orthogonal, this equals $\|Q_{\text{out}}^\top W X - (S\odot M) Q_{\text{in}} X\|_F^2$. For fixed rotations, the core update is therefore a standard 2:4 pruning problem in a rotated basis, and we solve it with the inverse-Hessian corrections of SparseGPT.

Wrapper sweeps. The wrappers are updated by DMRG-style two-site sweeps. All other sites are contracted into left and right "environments", which here are small matrices built from the data covariance $X X^\top$ and the current core. Two adjacent sites are merged and refit. Under the isometry constraint this refit is an orthogonal Procrustes problem, solved by one SVD. The merged tensor is then split back by SVD, keeping singular values above a threshold. That last step is the DMRG ingredient that ARMOR lacks: it sets the bond dimension separately at each cut from the singular value spectrum, instead of fixing one block size everywhere. Sweeps alternate with mask updates, in the spirit of ARMOR's block coordinate descent.

Channel ordering. Splitting $N$ channels into $n$ tensor sites requires choosing which channels sit next to each other. A bad choice creates artificial long-range correlation and inflates $D$. The MPO compression study of Gao et al. notes that this index grouping matters but leaves it to fixed heuristics. We test a reordering that clusters channels by activation correlation, analogous to the Jaccard row reordering of SMaT. Because this is a one-time static cost per layer, it avoids SMaT's preprocessing problem in dynamic settings.

Kernel. A chain of tiny contractions launched as separate operations will be bound by memory traffic and kernel launch cost. Gao et al. state that specialized kernels "may be necessary" and never measure latency. We will write one fused CUDA kernel that applies the $Q_{\text{in}}$ chain to tiles held in shared memory, feeds them to the 2:4 sparse tensor-core multiply, and applies $Q_{\text{out}}$ on the way out. ARMOR's own speed result comes from an unfused PyTorch prototype, so we build the same fused kernel for block-diagonal wrappers to keep the comparison fair.

Baselines

The naive reference points are the dense model and direct one-shot 2:4 pruning with SparseGPT, with no wrappers. On OPT-175B, SparseGPT at 2:4 raises WikiText-2 perplexity from 8.34 to 8.74 (SparseGPT). Gaps are larger on the 8–14B models we target, which is where wrappers matter.

The strongest competitor is ARMOR (ARMOR). It is the closest method in design and holds the best reported gap reduction in the file: on Llama-2-13B it cut the WikiText-2 perplexity gap "by nearly 50% compared to the next best baseline", and it scored 39.51 on GPQA with Qwen 2.5-32B against a dense 38.84. Its reported speed is 1.57× on a Qwen 2.5-14B layer, which is 84% of the theoretical 2:4 speedup. The file quotes no absolute pruned perplexity for ARMOR on Llama-2-13B, so we will reproduce it ourselves. We run ARMOR with the same 128 SlimPajama calibration samples it used, the same number of coordinate-descent iterations as our sweeps, and block sizes chosen to match our method's measured fused-kernel latency, not its parameter count. Beating ARMOR means a lower WikiText-2 perplexity and a higher zero-shot average at equal or lower measured latency, across at least two model families.

The second strong competitor is 3BASiL-TM. It adds a low-rank term to a 2:4 core and reports reducing the WikiText-2 gap to dense LLaMA-8B "by over 30%" with rank 64. We run it with its C4 calibration and choose the rank so its two-path kernel latency matches ours. Neither the file nor the paper reports inference latency for this method, so we measure it.

Two ablations isolate the key ingredient. In the first, we fix the rotations to MPOs with random orthogonal sites and the same bond dimensions, and prune only the core. This asks whether learned mixing matters or whether any cross-block rotation helps. In the second, we fix the bond dimension instead of letting SVD truncation choose it per cut. This tests whether the DMRG truncation step earns its place. Channel reordering is ablated on and off.

Why not just…?

Why not just use larger blocks in ARMOR? This is the real threat. If accuracy saturates once blocks reach a modest size, cross-block mixing buys nothing, and the MPO's extra kernel complexity is unjustified. The decisive test is the Pareto curve of perplexity against measured latency, sweeping ARMOR's block size and our bond dimension under identical fused kernels. If ARMOR's curve dominates, the proposal fails on its central claim, and we will report that outcome.

Why not just apply a fixed random or structured orthogonal rotation and then prune with SparseGPT? A fixed rotation spreads outlier weights across channels and costs nothing to fit. Our random-site MPO ablation is exactly this baseline at matched cost. If it closes most of the gap, the contribution shrinks to "rotation structure matters, fitting does not". The file contains no direct evidence either way.

Why not just compress the whole weight as an MPO, as in Gao et al., and skip sparsity? Gao et al. report up to 2000× compression in VGG-16 layers on CIFAR-10, at 93.76% versus 93.13% dense. However, they trained from scratch with backpropagation. Those results concern training from scratch, not post-training compression of pretrained language models, and they include no latency numbers. We include a pure-MPO post-training baseline at matched latency. We expect it to do poorly, because pretrained LLM weights lack the short-range correlation structure that small bond dimensions capture, but this has to be measured.

Why not just use an existing unstructured kernel such as Sputnik at 50% sparsity? Sputnik only beats dense cuBLAS from about 71% sparsity, and SMaT from about 78%. At 50% neither yields a speedup, so on Ampere-class hardware 2:4 is the only pattern that accelerates at this density.

Proposed Experiments

We compress Llama-2-13B, LLaMA-8B, and Qwen 2.5-14B, matching the models used by ARMOR and 3BASiL. We report WikiText-2 and C4 perplexity, the eight-task LM Harness zero-shot average, and GPQA for the largest model. Latency is measured on A100 GPUs as end-to-end tokens per second at batch sizes 1 and 16, plus per-layer microbenchmarks against ARMOR's 1.57× reference layer.

The first decisive check is the matched-latency Pareto curve against ARMOR described above. The second is a cheap mechanistic diagnostic run before the full study. For each layer, we compute the singular value spectrum across cuts of the optimally rotated weight. If the spectra show substantial weight beyond the block boundaries ARMOR can reach, cross-block mixing has room to help. If they decay within blocks, we expect a null result and stop early.

The third check is kernel efficiency. The fused MPO-wrapper kernel must reach at least ARMOR's 84% of the theoretical 2:4 speedup at the bond dimensions where accuracy improves. Otherwise the parameter savings never turn into wall-clock savings, and that is the failure the unstructured-sparsity literature keeps documenting.