When a large language model is compressed to two or three bits per weight, some groups of weights must get more bits than others. The principled way to choose is rate-distortion optimization: trade bits (the rate) against damage to the model (the distortion) with a single exchange rate, and give each weight group the bitwidth where its marginal damage per bit matches that rate. The fast versions of this idea assume each group's damage can be counted separately and summed. Meanwhile, the best low-bit quantizers do not quantize groups separately. They quantize one column, then nudge the not-yet-quantized columns to absorb its error. So the damage a column causes depends on which columns come after it and how much freedom they have to compensate. The allocator and the quantizer are describing two different processes.
This note proposes to make them describe the same one. Signal coding has a classical answer for sources whose components are correlated. If an encoder predicts each sample from its neighbours and codes only the prediction residual (predictive coding), bits should be allocated by the residual variance, not by the raw variance. Here the "residual" of a weight column is its sensitivity after later columns have compensated for it. This quantity is cheap: it falls out of the triangular factorization of the layer's Hessian, the matrix of second derivatives of the loss with respect to the weights, which second-order quantizers already compute. In these conditional coordinates, distortion becomes additive in expectation again, so the fast linear-time Lagrangian search survives unchanged. It now accounts for cross-weight dependencies, and no decorrelating transform has to be stored.
Core Claim
Let $H$ be a layer's proxy Hessian over its input columns, ordered as the quantizer will process them. Write $e_q$ for the rounding error of column group $q$ and $b_q$ for its bitwidth. Under sequential error feedback, the expected loss increase is approximately
$$ \Delta \mathcal{L} \approx \sum_q d_q \, \mathbb{E}[e_q^2], \qquad d_q = \frac{1}{[H_{F_q}^{-1}]_{qq}}, $$
where $F_q$ is the set of columns not yet quantized when $q$ is processed (including $q$). So $d_q$ is the Schur complement of $H$: the curvature that remains after the later columns have compensated. It equals a diagonal entry of an LDLᵀ factorization of $H$ in reversed order, and it never exceeds the marginal curvature $H_{qq}$.
With $\mathbb{E}[e_q^2] \propto s_q^2 2^{-2b_q}$, where $s_q$ is the group's range, the optimal allocation under an average budget $\bar b$ is reverse water-filling:
$$ b_q = \bar b + \tfrac{1}{2}\log_2 \frac{d_q s_q^2}{\big(\prod_{j} d_j s_j^2\big)^{1/N}}, $$
where $N$ is the number of groups. Bitwidths are then clipped to an allowed set such as {2, 3, 4} by a Lagrangian slope search.
Marginal allocation is the same formula with $H_{qq}$ in place of $d_q$. Classical predictive-coding theory says the gain of the conditional allocation over the marginal one grows with the spread between the two sets of quantities.
There is a corollary: processing order now matters for rate as well as for error. Columns quantized early can be compensated more, so they can be given fewer bits. We therefore treat column order and bit allocation as one joint choice.
The claim is deliberately narrow. Fast rate-distortion allocation fails at two bits mainly because its sensitivity measure is marginal while the quantizer is conditional. Fixing only the sensitivity measure should recover most of what heavier dependency models buy.
This also sharpens the additivity debate. Independent zero-mean rounding errors make the off-diagonal Hessian terms vanish in expectation. That is why the additivity used by RDO-Q is not simply wrong. The off-diagonal structure re-enters through compensation, and the conditional sensitivity $d_q$ is exactly where it lands.
Relation to Prior Work
Three lines of work supply the rate-distortion machinery. RDO-Q allocates channel bitwidths in CNNs with a linear-time search for the slope λ, relying on a Taylor-derived additivity of distortion. Radio extends rate-distortion allocation to LLMs: it derives logarithmic bit depths from gradient-variance sensitivity and solves the allocation by stochastic dual ascent. DeepCABAC uses a Fisher-weighted rate-distortion quantizer in front of a context-adaptive entropy coder.
All three measure each group's sensitivity marginally. We keep their Lagrangian structure and their cheap search, and replace the sensitivity with its conditional counterpart. Radio's log-sensitivity rule is the marginal special case of our formula.
Three lines of work motivate the gap by showing that dependencies matter. GuidedQuant shows that diagonal Hessian surrogates lose critical information and preserves within-channel dependencies under end-loss weighting. It works at a fixed nominal bitwidth and does no allocation. MG-PTQ runs a graph network over Hessian-weighted column graphs and allocates bits with Gumbel-Softmax. It captures interactions empirically, gives up the closed form and the duality, and assigns bits column by column, which is awkward for hardware. Quantization Error Propagation shows that cross-layer accumulation is what makes 2-bit layer-wise quantization collapse, and corrects for it with a propagation strength α.
We borrow QEP's insight for a second level of the method: layer-level conditional sensitivities computed on propagated rather than clean inputs. We do not propose a new compensation scheme.
Transform Quantization for CNN Compression constrains the design. It decorrelates weights with KLT or learned transforms and then allocates bits per transform channel, reaching 1–2 bits on ResNets, but it pays to store a transform per layer. Our method is its causal, triangular analogue. The decorrelation is carried out implicitly by error feedback, so nothing is stored, which directly addresses the transform-overhead problem that paper leaves open. A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization is a caution: vector quantizers dominate scalar ones at a matched rate. Our method improves allocation among scalar groups and composes with, rather than replaces, vector codebooks.
Research Program
The first piece of work is the allocator. For each linear layer of a transformer, we compute the proxy Hessian from calibration activations as GPTQ does, along with its reversed-order Cholesky factor. From these we read off $d_q$ for groups of 128 consecutive columns, so that bitwidths stay uniform within hardware-aligned blocks. Distortion is optionally reweighted by end-loss gradients following GuidedQuant's Fisher approximation.
A single global slope search across all layers then sets per-group bitwidths in {2, 3, 4}, or in {1, …, 4} for sub-2-bit budgets. The search has the same cost as in RDO-Q because the conditional distortions are additive.
The second piece is joint ordering. Starting from the activation-order heuristic, we greedily swap block order to maximize the predicted reduction in $\sum_q \log d_q$ weighted by allocated rate. We stop when the predicted gain falls below a threshold.
The third piece lifts the idea across layers. Layer-level slopes are recomputed with calibration inputs produced by the already-quantized prefix of the network, in the manner of QEP. A layer whose upstream noise can be absorbed by its own compensation then gets fewer bits.
The analysis component checks whether the theory predicts the gains. For each layer we compute the predicted coding gain of conditional over marginal allocation, which is the ratio of arithmetic-to-geometric-mean spreads of $d_q s_q^2$ versus $H_{qq} s_q^2$. We then correlate that prediction with the measured perplexity improvement across layers and models.
Baselines
The naive reference point is uniform-bitwidth GPTQ with the same group size and calibration set. It shows what allocation adds at all. MG-PTQ reports that GPTQ at 2 bits reaches 152.31 WikiText-2 perplexity on LLaMA-7b.
The most important comparison is marginal rate-distortion allocation in the style of Radio: log-sensitivity bitwidths from marginal Hessian or gradient-variance sensitivity, followed by exactly the same GPTQ-style compensated quantizer. This is the ablation that isolates our key ingredient, because the two runs differ only in $H_{qq}$ versus $d_q$. It runs on identical blocks, bit set, calibration set (128 C4 sequences) and slope search. We also run published Radio itself for reference. On a 3-bit Llama-2 70B, Radio reports 49.81% GSM8K accuracy against 46.47% for GPTQ. We found no reported Radio result on Lacuna for 2-bit Llama-2-7B WikiText-2 perplexity, so we will measure it.
The strongest dependency-aware allocator is MG-PTQ. It reports 39.53 WikiText-2 perplexity on LLaMA-7b at a 2.5-bit average. We will rerun it at matched average bits and matched calibration, both with its column-wise granularity and constrained to our 128-column blocks. Beating it means lower perplexity at equal average bits, and with far less allocation compute.
The strongest fixed-bitwidth dependency method is GuidedQuant. It reports reducing 2-bit Llama-2-7B WikiText-2 perplexity from 39.58 to 8.83. For cross-layer correction, the reference is QuIP+QEP, which QEP reports at 11.97 on the same setting.
Our method is an allocator, so the fair test against these is composition. We run GuidedQuant and QEP with and without our allocation at the same average rate, counting allocation metadata in the bit budget. A clear win is a consistent perplexity and zero-shot improvement at an identical average bit count.
Why not just…?
Why not just use marginal rate-distortion allocation, since compensation already handles dependencies? This is the most serious alternative. If compensation absorbs errors equally well under any allocation, marginal and conditional allocations will produce the same models. The ablation above settles it directly. The predicted-gain analysis tells us in advance which layers should differ: those where $d_q$ and $H_{qq}$ rank groups differently. If measured gains do not track predicted gains, the claim fails.
Why not just learn the allocation with a graph network, as MG-PTQ does? A learned allocator can in principle discover conditional effects. But its own ablation reports that replacing the graph network with an MLP hurts substantially, which suggests the useful signal is precisely Hessian structure. Our hypothesis is that $d_q$ is a closed-form summary of that signal. Matching MG-PTQ's perplexity with a single factorization and a slope search would show the learned part is unnecessary. If MG-PTQ wins clearly at matched block granularity, we report it as the method that captures something $d_q$ misses.
Why not just rotate the weights first so the Hessian becomes nearly diagonal? Incoherence processing of the kind QuIP uses spreads energy across coordinates. That could flatten the spread of $d_q$ and make both conditional and marginal allocation pointless. This is a real threat. We therefore run every comparison both with and without a random-rotation preprocessing step and report the predicted and measured gains in each condition. A shrinking gain under rotation would bound the method's scope honestly.
Why not just store a KLT and allocate in the transform domain? Transform Quantization reports a 4 dB learned-transform gain, but at the cost of per-layer transforms, which are large for LLM projection matrices. We include a KLT-domain allocator with transform storage counted in the rate. That shows whether the causal, storage-free version gives up much coding gain.
Proposed Experiments
The models are Llama-2-7B and 13B and Llama-3-8B, with OPT-125M through 6.7B for fast sweeps. We use average budgets of 1.8, 2.0, 2.5 and 3.0 bits. Calibration is 128 C4 sequences, with RedPajama as a second calibration set to probe calibration drift. Metrics are WikiText-2 and C4 perplexity, the zero-shot suite (BoolQ, PIQA, HellaSwag, WinoGrande, ARC-Easy, ARC-Challenge, OBQA), five-shot MMLU, and GSM8K on the 13B and 70B models at 3 bits against Radio's reported figure. We also report allocation wall-clock time and kernel throughput, since block-uniform bitwidths should run with standard mixed-block dequantization kernels.
The decisive checks are three. First, at 2.0 bits on Llama-2-7B, conditional allocation must beat marginal allocation under an otherwise identical pipeline, both alone and composed with GuidedQuant and with QEP. Second, the measured per-layer gain must correlate positively with the predicted coding gain. Third, joint ordering must add measurable improvement beyond activation ordering. If the first check fails, the paper becomes a well-controlled negative result: compensation already makes marginal allocation sufficient.
References
- RDO-Q: Extremely Fine-Grained Channel-Wise Quantization via Rate-Distortion Optimization
- Radio: Rate–Distortion Optimization for Large Language Model Compression
- DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
- GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
- Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models
- Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization
- Transform Quantization for CNN Compression
- A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization