Two training-free tricks make large language models cheaper to run, and each works well alone. The first is 2:4 sparsity, where every group of four consecutive weights keeps exactly two nonzeros. NVIDIA tensor cores can skip the zeroed weights in hardware. The second is 4-bit quantization of both weights and activations, which stores and multiplies numbers in four bits. For activations this only works after the model is "rotated": an orthogonal matrix is multiplied into the weights and inverted on the activations. This spreads out the few huge activation values that would otherwise ruin 4-bit rounding. The two tricks are widely believed to conflict, because a dense rotation mixes every input channel and so appears to erase any 2:4 pattern.
We argue the conflict comes from the order of operations, not from the math. If the model is rotated first and then pruned to 2:4 in the rotated coordinate system, the sparse weights stay sparse. The rotated activations simply feed a sparse matrix. The real open question is whether the rotated coordinate system is a good or a bad place to prune. Rotations that make activations easy to quantize also smear weight energy evenly across channels, and that is exactly what makes "keep the two largest of four" lossy. We propose to measure this tension directly. We then resolve it with block-diagonal orthogonal rotations, chosen on a small calibration set, that trade off outlier suppression against 2:4 pruning error and 4-bit rounding error jointly. The result would be the first training-free pipeline for models with 2:4 sparse weights plus 4-bit weights and activations.
Core Claim
The claim has two parts. The first is a measurable hypothesis. Under Hadamard or learned outlier-suppressing rotations, the error from 2:4 pruning rises relative to pruning in the original basis. This increase, more than any sparsity destruction, is the obstacle to stacking 2:4 sparsity with 4-bit activation quantization.
The second part is a method. A per-layer rotation $R$ is restricted to be block-diagonal and orthogonal, so it stays cheap to apply online and keeps the model's computational invariance. $R$ is chosen to minimize the output error of the fully compressed layer:
$$ \min_{R \in \mathcal{O}_b} \; \big\| W X - Q_w\!\big(P_{2:4}(W R)\big)\, Q_a\!\big(R^{\top} X\big) \big\|_F^2 . $$
The symbols are as follows. $W$ is the original weight matrix and $X$ the calibration inputs to that layer. $\mathcal{O}_b$ is the set of orthogonal matrices built from $b \times b$ diagonal blocks. $P_{2:4}$ is a one-shot 2:4 pruner with weight update, either SparseGPT-style or Wanda-style, applied along the input dimension. $Q_w$ and $Q_a$ are 4-bit weight and activation quantizers.
Rotations on the residual stream are folded into neighboring weights before any pruning happens, so they cost nothing at inference. Only the online rotations inside attention and the feed-forward block remain at runtime. Making those block-diagonal is what keeps them fusable with a sparse matrix multiply.
Relation to Prior Work
The method draws on two rotation-based quantizers. QuaRot shows that randomized Hadamard rotations enable end-to-end 4-bit weights, activations and key-value cache. SpinQuant learns the rotations on the Stiefel manifold, the set of orthogonal matrices. It does this with Cayley SGD on frozen weights, and reports that random rotations vary by up to 13 zero-shot points across seeds. We reuse SpinQuant's optimizer, but both papers treat dense matrices only. Neither asks how rotation interacts with a pruning constraint, and that interaction is our objective.
The hardware side constrains the design. Learning N:M fine-grained structured sparsity ties 2:4 patterns to sparse tensor cores. Work on unstructured sparsity, such as Wiedemann et al. and Grimaldi et al., shows that irregular zeros rarely beat dense arithmetic on GPUs. This is why we target 2:4 sparsity rather than unstructured sparsity.
Three papers sit closest to ours. ARMOR wraps a 2:4 core in block-diagonal matrices $A,B$ fitted by block coordinate descent, and its authors interpret these wrappers as moving weights into a basis where 2:4 hurts less. ARMOR is the strongest evidence that the choice of basis matters for 2:4 pruning. However, its wrappers are unconstrained, it stays in FP16, and it leaves quantization unstudied. Because its wrappers are not orthogonal, they also cannot serve as outlier-suppressing rotations for activation quantization.
SLiM combines 2:4 sparsity, 4-bit weights and a closed-form low-rank correction. It keeps activations in 16 bits precisely because it has no outlier mechanism. 3BASiL fits sparse-plus-low-rank decompositions in FP16 and lists quantization as future work.
Our proposal differs from all three in the same two ways. It quantizes activations, and it adds no extra low-rank branch, so it avoids the narrow-matrix latency penalty that A3 identifies. ARMOR's own measurements make this penalty concrete: its unfused wrappers achieve a 1.57x layer speedup, against 1.86x for plain 2:4.
Two benchmark and pruning results constrain the design. Benchmarking Composable Compression Techniques in MoE LLMs shows that stacked compressors degrade non-additively, and that pruning is the dominant failure mode. That result argues for optimizing the full compressed layer jointly rather than pipelining tools. SparseGPT and Wanda supply the pruner $P_{2:4}$.
Research Program
The work begins with a diagnostic study. On LLaMA-2-7B, LLaMA-2-13B and LLaMA-3-8B, we prune each linear layer to 2:4 with SparseGPT under four bases:
- the original basis;
- a QuaRot-style Hadamard basis;
- a SpinQuant learned basis;
- block-diagonal Hadamard bases with block sizes from 32 to full width.
For each basis we record three quantities: the relative 2:4 reconstruction error, the kurtosis of the activations (a measure of how heavy their outliers are), and the 4-bit rounding error. This study tests the hypothesis directly. It also shows whether the block size controls a smooth trade-off between pruning error and quantization error.
The method then optimizes the objective above layer by layer, using 128 calibration sequences from C4. The rotation is updated by Cayley SGD. The non-differentiable pruning mask and quantizers are handled by alternating two steps:
- fix the mask and pass gradients through the quantizers with a straight-through estimator;
- re-select the mask with the pruner under the new rotation.
Residual-stream rotations need a single shared rotation across layers to preserve the RMSNorm invariance that QuaRot relies on. We optimize this shared rotation against the summed error of the layers it feeds. To limit error accumulating from layer to layer, we optionally add a block-output matching pass in the spirit of 3BASiL's Transformer-Matching, applied to rotations only, with no weights trained.
Finally, we write a kernel that applies the online block-diagonal rotation and quantizes the activation tile in the prologue of a 2:4 sparse integer tensor-core matrix multiply. We target Ampere-class GPUs (A100, RTX 30/40), where sparse low-bit integer instructions exist. Where only sparse INT8 is available, we fall back to W8A8 at 2:4 sparsity.
Baselines
The naive reference point stacks off-the-shelf tools in the order practitioners would use. QuaRot rotates the model, SparseGPT prunes to 2:4 in that rotated basis, and GPTQ-style rounding then quantizes weights to 4 bits, with activations quantized to 4 bits as in QuaRot. The variant that prunes before rotating is also run, to confirm that it densifies the weights. Every pipeline uses the same models, the same 128 C4 calibration sequences, the same pruner, and the same total calibration GPU-hour budget, with matched hyperparameter search per method.
Two existing methods are the strongest competitors on specific aspects of the question. SLiM is the strongest prior 2:4 plus 4-bit pipeline. It reports up to 5.66% higher zero-shot accuracy on LLaMA-2-7B than earlier methods at 2:4 with 4-bit weights. We run it unchanged at W4A16 and also with 4-bit activations added, which is the condition it was not designed for.
SpinQuant is the strongest dense W4A4 method. It reports a 64.0 zero-shot average on LLaMA-2-7B (W4A4KV4), 2.9 points below full precision. This sets the accuracy ceiling for the quantization part alone. ARMOR followed by 4-bit quantization tests whether unconstrained wrappers do better than orthogonal ones; ARMOR reports a GPQA score of 39.51 against 38.84 dense on Qwen 2.5-32B in FP16. For the full combination of 2:4 sparsity with 4-bit weights and activations, no reported result was found on Lacuna, so the naive stack is the reference.
Beating the field means three things. First, lower WikiText-2 perplexity and higher zero-shot average than the naive stack and than SLiM with 4-bit activations, at equal sparsity and bit-width. Second, staying within roughly SpinQuant's dense W4A4 gap plus the 2:4 cost seen in FP16. Third, measured layer latency below dense W4A4. Two ablations isolate the key ingredient: the same block-diagonal rotations optimized without the pruning term, and random block-Hadamard rotations of matched block size.
Why not just…?
Why not just prune to 2:4 after a standard QuaRot rotation? This is the real threat. If the diagnostic study shows that Hadamard rotation barely raises 2:4 pruning error, the naive stack is nearly sufficient. In that case the contribution shrinks to an ordering result plus a fused kernel. We would report it as such and not oversell the optimizer. The head-to-head comparison against the naive stack, at matched calibration budget, settles the question.
Why not just skip activation quantization, as SLiM does? SLiM's W4A16 already yields large layer speedups: up to 4.3x on an RTX 3060 and 3.8x on an A100. It also reaches 0.23x the dense memory footprint. However, weight-only 4-bit still multiplies in 16 bits, so it gains little in compute-bound prefill and batched serving. We measure prefill latency and large-batch decode separately to show where 4-bit activations matter and where they do not.
Why not just use dense W4A4 and drop sparsity? QuaRot's fully 4-bit Llama-2-70B loses only 0.47 perplexity, and mature runtimes support it. If the 2:4 accuracy cost on 7B to 13B models exceeds what the sparse kernel gains in speed, dense W4A4 wins. We therefore report accuracy against measured latency, not against parameter count.
Why not just recover accuracy with a short LoRA fine-tune? That approach abandons the training-free setting, which is the deployment constraint here. Even so, we report a small LoRA recovery run as an upper reference. This shows how much of the remaining gap any training-free method can hope to close.
Proposed Experiments
The decisive checks come in order, and each answers a specific question.
First, the diagnostic: does 2:4 error increase monotonically as rotation block size grows, while kurtosis falls? A flat curve would weaken the motivation.
Second, the main comparison on LLaMA-2-7B, LLaMA-2-13B and LLaMA-3-8B. We measure WikiText-2 and C4 perplexity, and accuracy on the zero-shot harness (PIQA, WinoGrande, HellaSwag, ARC, LAMBADA, BoolQ). We also run MMLU and GSM8K, because the MoE benchmark found that multi-step reasoning breaks first under stacked compression. Results are averaged over three calibration seeds, given SpinQuant's finding on seed variance.
Third, the ablations on the pruning term and block size.
Fourth, kernel benchmarks on A100 and an RTX-class GPU. We report per-layer and end-to-end prefill and decode latency against three references: dense W4A4, dense FP16, and FP16 at 2:4 sparsity. We also report the fraction of the ideal 2:4 speedup retained, comparable to ARMOR's 84%.
The proposal succeeds if the optimized rotations close a measurable share of the gap between the naive stack and the dense W4A4 accuracy ceiling, at a latency below dense W4A4.
References
- QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs
- SpinQuant: LLM Quantization with Learned Rotations
- Learning N: M Fine-grained Structured Sparse Neural Networks From Scratch
- Dithered backprop: A sparse and quantized backpropagation algorithm for more efficient deep neural network training
- Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity
- ARMOR: High-Performance Semi-Structured Pruning via Adaptive Matrix Factorization
- SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
- 3BASiL: An Algorithmic Framework for Sparse plus Low-Rank Compression of LLMs
- A3: an Analytical Low-Rank Approximation Framework for Attention
- Benchmarking Composable Compression Techniques in Mixture-of-Experts LLMs
- SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot
- A Simple and Effective Pruning Approach for Large Language Models