When a large language model generates text one token at a time, the GPU spends most of its time reading weights from memory rather than doing arithmetic. Two compression tricks attack that cost from different directions. The first stores weights in 4 bits instead of 16. The second, activation sparsity, notices that many of the numbers entering each layer are near zero for a given token, so the weight columns they would multiply need not be read at all. Combining them should multiply the savings. In practice it mostly does not, for two reasons. Four-bit weights are stored in packed groups that share a scale factor, so skipping scattered individual columns saves almost no memory traffic. And when several users are served together, each user's zeros fall in different places, so almost every column is needed by someone and the skipping vanishes.
We propose to fix both problems with one offline change to the model: reorder the input channels of each layer so that channels which tend to be near-zero together sit next to each other. The reordering is mathematically free, since permuting a layer's weight columns together with the previous layer's outputs leaves the function unchanged. Once channels are arranged this way, the near-zero entries of a token land in whole contiguous blocks, and the block size can be chosen to equal the 4-bit quantization group. The GPU can then skip entire packed weight groups, scales included, with a coarse-grained kernel that runs on stock hardware. Because the reordering targets co-occurrence across many inputs, blocks that are idle for one request are also likely to be idle for the others in a batch. The work is a software-only hardware-software co-design: the unit of compression is matched to the unit of memory access on existing GPUs, and success is judged by measured bytes moved and wall-clock latency, not by counted operations.
Core Claim
The claim is that dynamic activation sparsity and grouped low-bit weight quantization compose multiplicatively on commodity GPUs only when the sparsity is coherent at the granularity of the quantization group. That coherence can be created offline by a learned channel permutation, with no retraining and no new silicon. The permutation should also delay the collapse of activation sparsity under batching that Training-Free Activation Sparsity in Large Language Models reports.
Write $m_b(x) \in \{0,1\}$ for whether block $b$ of the input vector is active for token $x$, where a block is active if its norm exceeds a calibrated threshold. For a batch $x_1,\dots,x_B$, the kernel must read the weight blocks in the union of the active sets. The fraction of weight bytes loaded is therefore
$$ \phi_B(\pi) = \frac{1}{K}\sum_{b=1}^{K} \mathbb{E}\left[\max_{i \le B} m_b(x_i)\right], $$
where $\pi$ is the channel permutation that defines the blocks and $K$ is the number of blocks. We choose $\pi$ to minimize $\phi_B$ on calibration data for a target batch size, subject to a quality constraint that is enforced through the threshold. Decode latency in the memory-bound regime should scale roughly with $\phi_B$ times the INT4 weight bytes, because weight reads dominate decoding time. The first thing to verify is how closely measured latency follows $\phi_B$.
Relation to Prior Work
The evidence that a gap exists comes from four papers. Training-Free Activation Sparsity in Large Language Models zeroes low-magnitude activations with a custom sparse matrix-vector kernel. It gets real speedups at batch size one, but notes that the speedups degrade as batch size grows because sparsity patterns diverge across tokens. DuoGPT combines unstructured weight pruning with activation sparsity and claims compatibility with 4-bit quantization, but its results rest on software emulation rather than a kernel that exploits both. Motivating Next-Gen Accelerators with Flexible N:M Activation Sparsity shows that activation sparsity beats weight sparsity at equal ratios and that coarser patterns such as 16:32 hurt accuracy far less than 2:4. It concludes that new accelerators are needed, and it observes a collapse on the generative benchmark IFEval. We argue that coarse, block-shaped sparsity can be made cheap on existing hardware instead.
Other work supplies methods but also constrains the design. Celty gets strong dual-sparse decode speedups with a compressed sparse format, but it relies on new GPU microarchitecture (a hardware decoder and register-file accumulators), does not integrate weight quantization, and reports that software index reconstruction wastes much of the execution time. That observation is exactly why we avoid fine-grained indexing. TileMix shows that routing computation at hardware-aligned tile granularity inside a fused kernel pays off on A100s, though its routing is static. We borrow the tile-alignment principle and make the routing dynamic.
The hardware co-design line (SpAtten, TENET, the ADMM memristor framework) matches compression structure to hardware structure by building the hardware; see SpAtten, TENET, and the ADMM memristor framework. We keep the hardware fixed and reshape the model's data layout to fit it.
Two papers affect how we read results. Unified Scaling Laws for Compressed Representations finds that the quality costs of quantization and sparsity multiply when both are applied. We use this as a null hypothesis for quality: if permutation-induced block sparsity plus INT4 loses more than the product of their separate retentions predicts, the two interact badly. FPSAttention warns that quantization error concentrates on exactly the large values that sparsity preserves. We avoid that tension by quantizing only weights and keeping activations in 16-bit.
Research Program
The program has three parts.
The first part is the permutation. For each linear layer we collect activation magnitudes on C4 calibration samples, following the 128-sample protocol of DuoGPT. From these we build a channel co-activation statistic and solve for a permutation that packs co-inactive channels into blocks of size $g$, the quantization group size (64 or 128). The objective is $\phi_B$ above. The search is a balanced clustering problem, and we will compare greedy agglomeration against a spectral relaxation. Permutations propagate through residual streams, so we permute only inputs whose producers can absorb the permutation: the MLP down-projection, whose input is the MLP's hidden activation, and the attention output projection. Inputs read directly from the residual stream stay unpermuted, and we report how much of the model is covered.
The second part is the kernel. It is a Triton INT4 matrix-vector and small-batch matrix kernel. It computes block norms of the input and, per batch, the union mask over blocks. It then loads only the packed groups and scales of active blocks, contiguously, with no per-element indices.
The third part is the analysis. We measure bytes actually read and latency against the predicted $\phi_B$ across batch sizes and hardware, and test the multiplicative-quality null hypothesis.
Baselines
The naive reference point is dense FP16 decoding with a standard fused kernel. The second reference is INT4 weight-only group quantization with no sparsity, using the same group size and the same kernel family with skipping disabled. This is the baseline the proposal must beat most clearly. If activation sparsity adds nothing on top of 4-bit weights, the idea is wrong. We found no reported result on Lacuna for the speedup of INT4 weight-only decoding, so we will measure it ourselves on identical hardware.
The strongest existing software method is TEAL (Training-Free Activation Sparsity in Large Language Models). It reports a 1.53× speedup at 40% sparsity and 1.8× at 50% on an NVIDIA A6000, with a LLaMA-3-8B downstream average of 66.21 against 68.07 dense. We will run TEAL both in FP16, as published, and stacked on the same INT4 weights, on LLaMA-3-8B and the same A6000, with the same calibration data. We will match quality: each method's threshold is tuned so that WikiText-2 perplexity sits within the same tolerance of the INT4 dense model. Beating TEAL means lower measured decode latency at matched quality at batch size one, and a clearly slower decay of speedup at batch sizes 4 to 16.
DuoGPT (DuoGPT) reports 72.56% average zero-shot accuracy on LLaMA-3-70B and a 1.39–1.51× iso-speedup. We will compare accuracy at matched sparsity on LLaMA-3-8B, where our compute budget allows, and we will note that its speedups come from emulation.
Celty (Celty) reports a 2.63× decode speedup on LLaMA-2-13B at 60% dual sparsity, with WikiText-2 perplexity rising to 7.2 from 4.9. It needs modified hardware, so we will compare only against its software kernel's reported regime and its perplexity-speedup trade-off on LLaMA-2-13B. We will not claim a like-for-like latency comparison.
Two ablations isolate the key ingredient. The first uses block-level skipping with the identity permutation, which separates the effect of reordering from that of coarse skipping. The second uses a random permutation, which rules out any benefit from merely breaking the original channel order.
Why not just…?
Why not just use 4-bit weights and stop there? In memory-bound decoding, 4-bit weights already cut weight traffic about fourfold, and this is a real threat to the proposal. If activation sparsity at matched quality removes only a small fraction of blocks, the added kernel overhead could cancel the gain. The INT4-dense baseline exists to settle this. The proposal succeeds only if the measured $\phi_1$ at matched perplexity falls well below one and latency follows it.
Why not just stack TEAL's element-level sparsity on INT4 weights? Packed 4-bit groups share a scale and are fetched as contiguous words. Skipping scattered elements therefore saves little traffic, and it requires the index bookkeeping that Celty identifies as a major software cost. This stacked configuration is run directly as a baseline, so the claim can be falsified.
Why not just statically prune the channels that are always near zero? A permutation that clusters co-inactive channels might simply rediscover dead channels. In that case plain structured pruning would match it with a simpler dense kernel. We will report the fraction of skipped blocks that are input-dependent, meaning they are skipped for some tokens but not others, and we will compare against static channel removal at the same byte budget. Motivating Next-Gen Accelerators and DuoGPT both find that dynamic sparsity retains more accuracy than static pruning. We expect the same here but will test it rather than assume it.
Why not just take the union mask across the batch without permuting? This is the identity-permutation ablation run at $B>1$. TEAL's reported batch collapse suggests the union becomes nearly dense quickly without coherence. Whether permutation changes that is the central empirical question.
Proposed Experiments
We will use LLaMA-3-8B, to match TEAL, and LLaMA-2-13B, to match Celty, both with INT4 group quantization at group sizes 64 and 128, on an NVIDIA A6000 and one smaller consumer-class GPU. Batch sizes are 1, 2, 4, 8, and 16. Quality is measured with WikiText-2 perplexity and the zero-shot suite of MMLU, ARC, HellaSwag, PIQA, WinoGrande, and BoolQ. Because Motivating Next-Gen Accelerators found that activation sparsity collapses on generative tasks, IFEval is a required generative check. LongEval tests whether block sparsity damages retrieval over long contexts. Cost is measured as bytes read per token from hardware counters, per-token decode latency, and throughput at each batch size.
The first decisive check is whether measured latency tracks the predicted $\phi_B$ across batch sizes. If it does not, kernel overheads dominate and the byte-count model is wrong. The second is whether permuted block sparsity beats INT4-dense latency at batch size one while staying within tolerance on IFEval, not just on the multiple-choice suite. The third is whether the batch size at which speedup falls below 1.2× moves substantially later with the learned permutation than with the identity or a random one. The fourth is whether combined quality loss matches the product of the separate quantization and sparsity retentions, as Unified Scaling Laws for Compressed Representations would predict, or falls below it, which would expose an interaction. A negative result on the third check alone would still leave a useful single-user edge result. A negative result on the second would mean the idea does not hold.
References
- Training-Free Activation Sparsity in Large Language Models
- DuoGPT: Training-free Dual Sparsity through Activation-aware Pruning in LLMs
- Motivating Next-Gen Accelerators with Flexible N: M Activation Sparsity via Benchmarking Lightweight Post-Training Sparsification Approaches
- Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference
- TileMix: Tile-Centric Mixed-Precision Attention for LLM Inference Acceleration
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
- TENET: An Efficient Sparsity-Aware LUT-Centric Architecture for Ternary LLM Inference On Edge
- An Ultra-Efficient Memristor-Based DNN Framework with Structured Weight Pruning and Quantization Using ADMM
- Unified Scaling Laws for Compressed Representations
- FPSAttention: Training-Aware FP8 and Sparsity Co-Design for Fast Video Diffusion