Spending Idle Compute to Climb the 2-Bit Wall

When a language model writes text one token at a time, the GPU spends most of its time waiting on memory rather than doing arithmetic. Every generated token requires streaming all of the model's weights from off-chip memory, and the arithmetic units sit partly idle. Quantization helps because it stores weights in fewer bits. Quantization means rounding each weight to a small set of values, for example 2 bits instead of 16. At 2 bits, though, accuracy collapses. The standard repair adds a small low-rank correction matrix on top of the rounded weights, and at 2 bits that correction needs so many dimensions that its own size cancels the savings. LQER calls this the "2-bit wall" (LQER).

This note proposes building the correction to be cheap in bytes moved rather than cheap in arithmetic, because bytes are what limit decoding speed. The correction is factored into a shared basis and small per-layer coefficients. The shared basis is reused by every projection that reads the same input, and possibly by all layers, so it can stay resident in on-chip cache. Only the coefficients are stored in low precision and streamed from main memory. The extra multiply-adds run on arithmetic units that are otherwise waiting. If correction directions really are shared, a byte budget that buys a rank-32 correction today could buy a correction several times larger. That would make 2-bit bases usable without irregular memory access and without lookup tables.

Core Claim

AI and Memory Wall supplies the motivation. It shows that decoders run matrix-vector products whose speed is set by memory operations, and that compute has grown about 3.0x every two years while DRAM bandwidth has grown about 1.6x. Low-rank error correction should therefore be priced in bytes moved per weight, not in floating-point operations. Under that pricing, the expensive part of LQER's correction is storing two 16-bit factors per layer, not the rank itself.

For a weight matrix $W$ with $m$ outputs and $n$ inputs, we approximate $W \approx Q + U V^\top$. Here $Q$ is the $b_Q$-bit base, $U$ ($m \times r$) holds per-matrix coefficients at $b_U$ bits, and $V$ ($n \times r$) is a basis at $b_V$ bits shared by $G$ matrices. The effective storage per weight is then

$$\bar b = b_Q + \frac{r\,b_U}{n} + \frac{r\,b_V}{G\,m}.$$

The formula comes from dividing each factor's total bits by $mn$. LQER's unshared 16-bit factors correspond to $b_U=b_V=16$ and $G=1$.

As an illustration, take a 4096×4096 matrix with $b_Q = 2$. Rank-64 LQER-style factors add about 0.5 bits per weight. Coefficients at 4 bits with a widely shared basis add about 0.06 bits per weight. That frees roughly an eightfold larger rank at equal bytes.

The scientific bet is that the shared basis is not a free lunch. It works only if quantization error directions are shared across matrices. There is a specific reason to expect this. LQER's activation-scaled variant, $L^2$QER, weights the error by input activation statistics. Every projection that reads the normalized residual stream sees the same high-magnitude input channels. The dominant right-singular directions of the scaled error should therefore cluster on those channels. Within a Transformer block this sharing is almost guaranteed for the query, key, value, gate, and up projections, which read an identical input. The bolder and more useful version is sharing across layers.

Research Program

The first stage is purely measurement. On LLaMA-2-7B, Llama-3.1-8B, and LLaMA-2-70B, we quantize weights to 2-bit and 3-bit bases with GPTQ-style rounding and compute the activation-scaled error for every linear layer. We then measure how much of each layer's error energy, in the activation-weighted norm $L^2$QER uses, is captured by a rank-$r$ basis. We compare three choices of basis: one fitted to that matrix alone, one shared within a block among projections that read the same input, and one shared across all layers of the same input type. The output is a curve of captured energy against rank for each sharing level. That curve decides whether the rest of the program uses cross-layer sharing or falls back to within-block sharing.

The second stage builds the method, which needs no training. We fit $V$ by a truncated SVD of the stacked, activation-scaled errors of the $G$ sharing matrices. We solve for each $U$ in closed form given $V$, then quantize $U$ to 4 bits with per-group scales. We then re-solve $Q$ by rounding $W - UV^\top$ again, for a few alternations. The alternation is cheap, and it helps because the base and the correction should not both try to fix the same error.

A further step addresses compounding. Layer-by-layer allocation ignores how error builds up across layers, a limitation noted for MxMoE. The total rank budget is therefore split across sharing groups using the change in end-of-model loss measured on calibration data, not per-layer reconstruction error. The split is a one-dimensional greedy search over a handful of rank levels per group, which keeps calibration cost close to LQER's reported 1.2 hours on one A100 for LLaMA-33B.

The third stage is the kernel. The decode step computes $Qx + U(V^\top x)$. The product $V^\top x$ is computed once per shared input, from a basis kept in L2 cache, and is broadcast to all $G$ matrices. The main kernel then streams the packed 2-bit $Q$ and 4-bit $U$ in contiguous tiles and dequantizes in registers. The correction is fused into the same pass, so the output is written once. Every access is a dense, aligned tile, which follows the design rule LQER uses to avoid the scatter-gather stalls of outlier-splitting schemes.

This design avoids lookup tables, unlike FluxBin. FluxBin precomputes dot products for binary weights in shared memory, and its table size grows as $2^\mu$ with sub-vector length $\mu$. It needs no new hardware format, unlike MixFP4. It runs on Ampere and Ada GPUs, unlike MicroMix, whose gains depend on Blackwell FP4 units.

There is one direct conflict to address. QuaRot uses random Hadamard rotations to spread outliers evenly across all channels. Spreading the outliers should also spread the error spectrum and weaken the shared-channel structure the method relies on. We do not avoid this. Every configuration is run with and without rotation. If rotation destroys the sharing, that finding is part of the contribution, and it bears on the open question of whether rotation removes the need for heterogeneous formats. MixFP4's evidence is that block-level heterogeneity survives rotation, which suggests some structure may remain.

Baselines

The naive reference points are the FP16 model and plain 2-bit and 3-bit GPTQ-style weight quantization with no correction. They anchor how far the 2-bit cliff falls and how much an extra base bit buys.

The most direct competitor is LQER with its $L^2$QER scaling (LQER). It is the closest existing method because it already performs analytical low-rank error correction on regular blocked matrix multiplications. We run it on the same calibration set at a 2-bit base, with its ranks chosen so that its effective bits per weight match each of our configurations. LQER reports downstream accuracy within 0.3% of FP16 on LLaMA-2-70B with a 4-bit base. No reported LQER result at a 2-bit base was found on Lacuna, so we produce that number ourselves. To beat LQER, we must give lower WikiText-2 perplexity and higher zero-shot accuracy at the same effective bits, measured on the same GPU.

The strongest system-level competitor in the ultra-low-bit regime is FluxBin. On A100 it reports a 5.92x speedup over FP16 on LLaMA-2-7B and runs 3.25x faster than QuIP# on 70B models. We compare against it at matched model size and matched memory footprint. We measure both its accuracy and its decode throughput on our hardware. We also test it on a GPU with less shared memory, because its table-size limit predicts it will degrade there. To beat FluxBin, we must match its accuracy at equal bytes and match or exceed its throughput, with the advantage widening on smaller-cache GPUs.

For the mixed-precision family we use MicroMix's channel-reordered allocation, restricted to integer formats so it runs on pre-Blackwell hardware, at matched average bits. MicroMix reports near-FP16 accuracy at about 5.2 bits on Qwen2.5-32B. The comparison asks whether spending extra bytes on correction beats spending them on promoting sensitive channels.

Two ablations isolate the key ingredient. The first quantizes LQER's factors to 4 bits but does not share $V$. This separates the gain from low-bit factors from the gain from sharing. The second shares $V$ within a block but not across layers. All methods get the same 128-sequence calibration set and the same alternation budget, and every reported speed comes from a fused kernel rather than a simulation.

Why not just…?

Why not just use a 3-bit base and skip the correction? At about 2.3 to 2.5 effective bits, this is the most honest competitor. A correction is only worth it if its fraction of a bit buys more accuracy than the same fraction spent on the base. The first experiment compares the two directly on the same curve: perplexity against effective bits, with the 2.0, 2.25, 2.5, and 3.0-bit uniform points marked. If uniform 3-bit dominates at matched bytes, the method has failed at its stated goal and we will report that.

Why not just quantize LQER's factors and stop there? This is a real threat, because low-bit factors alone already shrink the correction's size by about 4x. The first ablation above measures it directly. If sharing $V$ adds nothing beyond quantized unshared factors, the proposal reduces to a simpler and still useful result. We would publish it as such and drop the cache-resident basis.

Why not just fine-tune with LoRA (small trainable low-rank adapters) or train with quantization in the loop? These methods recover more accuracy at much higher cost. Restoring Pruned Large Language Models via Lost Component Compensation reports a 2.01% mean accuracy gain over the best baseline when correction targets the lost subspace instead of searching blindly. LQER reports 1.2 versus more than 7 hours of calibration against iterative alternatives. Our claim is restricted to the training-free regime. We report one LoRA-recovered 2-bit model as an upper reference, not as a target to beat.

Why not just rotate and quantize uniformly, as QuaRot does? QuaRot reports a 0.47 perplexity increase on Llama-2-70B at 4 bits for weights, activations, and the cache. That is a different, compute-oriented regime. Its 2-bit weight behavior is the open question. We include rotated 2-bit uniform quantization as a baseline, and rotated variants of our own method, so the experiments show whether rotation and shared correction are complements or substitutes.

Proposed Experiments

The decisive first check is the shared-basis energy curve from stage one. Suppose a cross-layer basis of rank $r$ captures much less activation-weighted error energy than per-matrix bases of rank $r/4$, which is roughly what equal bytes allow. Then cross-layer sharing is abandoned and the program proceeds with within-block sharing only, where the shared input makes sharing much more likely to hold.

The accuracy evaluation uses WikiText-2 and C4 perplexity, plus the zero-shot suite of PIQA, ARC-Easy, ARC-Challenge, BoolQ, HellaSwag, WinoGrande, and LAMBADA. It also includes MMLU and GSM8K, because LLMCBench finds that compression damages factual knowledge and reasoning differently (LLMCBench). Every result is plotted against effective bits per weight, including all scales and factors.

The systems evaluation measures single-batch and small-batch decode throughput on A100 and RTX 4090. It also profiles HBM bytes per token, to confirm that the shared basis is actually served from L2 cache rather than reloaded from main memory. The last check sweeps batch size until decoding becomes compute-bound, to locate the point where the extra multiply-adds stop being free. Past that point, the method's advantage should disappear in a way the byte-based accounting predicts.

The proposal succeeds if, at about 2.25 to 2.5 effective bits, it beats both bit-matched LQER and uniform 3-bit quantization on perplexity and on GSM8K, while staying within 10% of plain 2-bit decode speed.