Pricing Compression in Milliseconds: Latency-Measured, Layer-Adaptive Choice of Compression Format for LLMs

Today, compressing a large language model means picking one tool and applying it everywhere. Every weight matrix is quantized to 4 bits, or every matrix is pruned to the same sparsity pattern, or every matrix is cut to the same rank. Two facts make this wasteful. First, layers differ greatly in how much damage they tolerate. Query and key projections can lose most of their rank, while value and down projections cannot. Second, the savings a method promises on paper (fewer parameters, fewer floating-point operations) often do not show up as faster inference on a real GPU. We propose to choose, separately for each weight matrix, which compression format to use and how aggressively to apply it. Each option is priced by its measured latency on the target GPU, not by its parameter count, and the choice is made by solving a small budgeted selection problem.

The bet is concrete. Formats have very different real costs. Two-out-of-four structured sparsity (keeping 2 of every 4 weights in a pattern NVIDIA hardware can accelerate) has weak runtime support. A low-rank factorization replaces one matrix multiply with two smaller ones and pays an extra kernel launch. A quantization-error correction term adds a side multiply. Layers also differ in how much accuracy each format costs them. So the best mixture of formats under a latency budget should differ from both a uniform recipe and from allocations tuned for parameter count. It should also differ between the token-by-token generation phase and the prompt-processing phase, because these behave differently on the same GPU.

Core Claim

At a matched, measured end-to-end latency on a given GPU, a per-matrix choice among a small menu of formats will give lower perplexity and higher zero-shot accuracy than the strongest uniform multi-format method and the strongest parameter-budgeted rank allocator. The menu contains FP16 kept dense, INT4 weight-only quantization, INT4 plus low-rank error correction, 2:4 sparsity plus INT4 plus low-rank adapter, and truncated SVD with rank rounded to a multiple of the GPU tile size. The choice is made by multiple-choice knapsack, using per-(matrix, format) accuracy costs and per-(shape, format, regime) latency tables.

A second, falsifiable claim is that the optimal assignment depends on regime. An allocation solved for single-sequence decoding should be measurably worse when deployed for large-batch prefill, and the reverse. If that is true, "compression ratio" is not a property of the model alone.

Written out, for each weight matrix $l$ we choose a configuration $c_l$ from the menu $\mathcal{C}$ to solve

$$\min_{\{c_l\}} \sum_l s_l(c_l) \quad \text{s.t.} \quad \sum_l t_l(c_l; \rho) \le T,$$

where $s_l(c)$ is the increase in calibration loss when only matrix $l$ is compressed with configuration $c$, $t_l(c;\rho)$ is the measured kernel time of that matrix's shape in format $c$ under deployment regime $\rho$ (batch size and phase), and $T$ is the latency budget.

Relation to Prior Work

Three groups of papers shape this proposal, each in a different role.

Papers that motivate it by documenting layer heterogeneity. SVD-LLM V2 allocates SVD truncation non-uniformly from truncation-loss profiles and finds roughly tenfold differences in loss across layers at equal compression. ARA learns rank allocations with a monotone staircase mask and confirms that query/key projections are compressible while value/down projections are not. LLRC learns which singular values to keep with Gumbel-Sigmoid masks. All three spend a parameter budget on a single format (rank). We spend a latency budget across several formats. Their sensitivity findings are the reason we expect non-uniform choices to matter.

Papers that supply the menu items. SLiM combines convex 4-bit quantization, 2:4 sparsity and closed-form saliency-weighted low-rank adapters. LQER corrects quantization error with an activation-scaled low-rank term $W_q + AB$. ARMOR wraps a 2:4 sparse core in block-diagonal correctors. 3BASiL fits sparse-plus-low-rank decompositions with ADMM. Each applies one structure uniformly: fixed 2:4 everywhere, fixed rank 64, or fixed rank $k$. LQER names layer-wise rank selection as future work. We reuse their per-matrix fitting procedures unchanged and contribute only the allocation across them. Isolating the allocation is what lets us attribute any gain to it.

Papers that constrain the design by showing paper savings do not equal speed. ShuffleNet V2 and EfficientNet-X show that memory traffic and operational intensity (arithmetic per byte moved), not operation counts, govern runtime. Polyline Path Masked Attention shows fragmented operations lose their savings to kernel launches. LLMCBench reports that 2:4 sparsity lacks the mature runtime support of INT4/INT8. These are why the cost side of our objective is measured on hardware rather than computed.

The closest precedents for joint, hardware-priced allocation are DJPQ and Balaskas et al.. DJPQ learns pruning and bit-widths jointly under a bit-operations penalty. Balaskas et al. use reinforcement learning against an Eyeriss energy model. Both work on CNNs and use abstract or simulated costs, and neither includes low-rank formats. DJPQ's finding that heavily pruned layers receive more bits warns us that format choices interact, which we test directly.

Research Program

The work has four parts, each of which yields a result on its own.

Latency atlas. For every distinct weight shape in LLaMA-2-7B and LLaMA-3-8B, we benchmark each menu format on an A100 and an RTX 3090. We do this at decode batch sizes 1 and 16 and at prefill with 2048 tokens. We use the best available kernels for each format: existing INT4 kernels, the SLiM and ARMOR implementations, and two-GEMM low-rank with and without a fused kernel.

The atlas is useful on its own. It tells us, per shape and regime, where measured latency departs from the parameter-byte prediction. We expect the largest departures for 2:4 cores, for unaligned ranks, and for the extra GEMM in error-corrected formats.

Sensitivity table. For each matrix and each format at several strengths (ranks, adapter sizes), we compress that matrix alone. We record the rise in loss on 128 calibration sequences, following ARMOR's setup on SlimPajama. A second disjoint split is held out to detect overfitting to calibration data.

Selection and correction. We solve the knapsack exactly by dynamic programming over discretized latency, which takes seconds. Because single-matrix sensitivities may not add up, we then run one or two correction passes. Each pass re-measures loss with the full assignment in place, re-estimates costs for matrices in the most damaged blocks given their neighbours' formats, and re-solves.

Analysis. We report how the chosen formats distribute over projection types and depth, and how they shift between regimes and GPUs. We also test whether the knowledge-versus-reasoning asymmetry reported by LLMCBench emerges: does the solver send quantization to some matrices and sparsity to others in a way that tracks those capabilities?

Baselines

Naive reference points. The first is the dense FP16 model. The second is uniform INT4 weight-only quantization (GPTQ-style), the most deployable single tool. Both are run on the same GPUs with the same timing harness. They anchor every other method in milliseconds and in accuracy.

Strongest uniform multi-format method: SLiM. SLiM applies 2:4 sparsity, 4-bit quantization and a closed-form adapter to every layer. It reports up to 5.66% higher downstream accuracy than prior one-shot 2:4+4-bit methods on LLaMA-2-7B and a 3.8× layer-wise speedup on A100. No reported end-to-end latency-matched perplexity result was found on Lacuna for SLiM, so we will measure it ourselves. It is the strongest competitor because it already combines all three levers and runs fast. If our allocator cannot beat it at its own measured latency, non-uniformity buys nothing. We run SLiM with its released settings and the same calibration samples. Its measured end-to-end latency becomes the budget $T$ for our solver. Beating it means lower WikiText-2 and C4 perplexity and higher average zero-shot accuracy at equal or lower measured latency, consistent across three calibration seeds.

Strongest non-uniform allocator: ARA followed by INT4. ARA learns per-layer ranks and is shown to compose with 4-bit GPTQ. On LLaMA2-7B at an 80% compression ratio it reduces WikiText-2 perplexity from 8.38 (uniform) to 6.42 and raises average zero-shot accuracy by 9.72 points over uniform compression. It also reports a 1.26× throughput gain on RTX 3090 at 60% compression. We run ARA with its own training budget, quantize to INT4, and tune its compression ratio until its measured latency matches $T$. We also include SVD-LLM V2 plus quantization, which reports 1.29×–2.71× A100 throughput across 20–80% compression, as a second non-uniform reference. Beating these means better accuracy at matched milliseconds rather than at matched parameters.

Ablations that isolate the key ingredients. The first replaces measured latency $t_l$ with parameter bytes, keeping everything else fixed. This isolates the value of the latency atlas. The second restricts the menu to a single format family (rank only, or INT4 with error-correction rank only) while keeping the latency pricing. This isolates the value of mixing formats. The third skips the correction passes. This isolates how much non-additivity matters.

Why not just…?

Why not just quantize everything to INT4? This is the most serious threat. INT4 is well supported, and LLMCBench reports that quantization keeps over 95% of original performance. In the memory-bound decode regime, INT4 may already sit near the latency floor for most matrices. There is contrary evidence: SVD-LLM V2 reports that SVD followed by 4-bit quantization beats direct 3-bit quantization at equal memory, which suggests mixing formats helps once budgets go below uniform INT4. Our experiments settle this by sweeping budgets from INT4's own latency downward. If the allocator only wins below that point, we will report the method as a tool for sub-INT4 budgets, not a general replacement.

Why not just use parameter bytes as the cost, since decode is memory-bound? If bytes predicted latency well, the atlas would be redundant. The bytes-cost ablation tests exactly this. The file gives reasons to expect departures. LLMCBench notes 2:4 sparsity's weak runtime acceleration. ARMOR's measured speedup of 1.57× sits below the 1.86× of naive 2:4. Two-GEMM low-rank layers pay kernel-launch and alignment costs. If the bytes ablation matches the full method in decode, we will narrow the claim to prefill and large-batch serving, where compute-bound behaviour makes bytes a poor proxy.

Why not just run ARA or SVD-LLM V2 and then quantize? These already allocate non-uniformly and compose with quantization. But they spend a parameter budget on one format. They cannot choose 2:4 cores for matrices whose shapes accelerate well, or leave a sensitive down projection dense when that is cheap in milliseconds. The matched-latency comparison above is the direct test. A win only on the parameter axis would not count.

Why not just learn the allocation end to end, as DJPQ does? A differentiable joint search could capture interactions between formats that our additive costs miss. However, DJPQ optimizes an abstract bit-operations cost that ignores bandwidth and launch overhead, and it was shown only on CNNs. Gradient-based relaxations over discrete kernel formats also cannot be priced with measured latency without a surrogate model. Our correction-pass ablation measures how much interaction actually costs. If the gap from skipping correction is large, a learned joint method becomes the natural next step.

Proposed Experiments

Models and evaluation. We compress LLaMA-2-7B and LLaMA-3-8B, evaluating WikiText-2 and C4 perplexity, the zero-shot suite (BoolQ, PIQA, ARC-Easy/Challenge, WinoGrande, OpenBookQA), MMLU, and GSM8K. Latency is end-to-end per-token decode time and prefill time on A100 and RTX 3090, measured with one harness for all methods.

Matched-latency comparison. The main result is accuracy against measured latency at three budgets: SLiM's latency, uniform INT4's latency, and a budget midway between them. All methods are compared against all baselines at each budget.

Decisive check one: regime dependence. We solve allocations for batch-1 decode and for 2048-token prefill, then deploy each in the other regime. The claim fails if cross-deployment costs less than the seed-to-seed variance.

Decisive check two: hardware dependence. We transfer the A100 allocation to the RTX 3090 and compare it against the allocation solved natively for the 3090.

Decisive check three: calibration shift. We compare the held-out calibration loss against downstream scores. This checks whether the solver overfits its 128 sequences, a known weakness of rank learners that rely on small WikiText-2 or C4 subsets.

What would kill the idea. If the bytes-priced ablation and ARA-plus-INT4 both match the full method within noise at every budget and regime, the latency atlas and the multi-format menu add nothing. We would report that as a negative result about where real speed comes from in LLM compression.