Information Loss, Not Distortion, Predicts Which Quantization Damage Can Be Repaired

When a large language model is compressed to 2 bits or fewer per weight, its accuracy does not decline gradually. It falls off a cliff. Much current practice then tries to repair the damage with light fine-tuning, such as adjusting scales and codebooks or training small adapters. Sometimes this works and sometimes it does not. We propose a simple rule for telling the two cases apart. Quantization error that merely distorts a layer's activations can be undone by later computation or cheap fine-tuning. Quantization error that destroys information cannot be undone. Only the second kind should receive the extra bits.

To act on this rule we need a measurement that separates the two cases, and the usual one, mean-squared error between full-precision and quantized activations, does not. A layer can show large error that a decoder inverts easily, or small error that erases a direction the network relies on. We define a layer's information deficit: the number of bits about the input that the full-precision layer carries and the quantized layer has lost, counted only at the resolution the rest of the network actually uses. We estimate it with a learned-decoder upper bound that stays tractable at 4096 dimensions. We then test whether the deficit predicts post-repair accuracy, and whether allocating bits by deficit beats Hessian- and spectrum-based allocation at a fixed 2-bit budget.

Core Claim

Below 3 bits, the layers of a quantized LLM split into two groups. Some are distorted but still informative: their error is large, yet a decoder recovers the full-precision activation from the quantized one. Others destroy information. Our hypothesis is that the second group explains most of the accuracy that survives fine-tuning uncorrected, and in particular the collapse of multi-step reasoning while fluency is preserved. An empirical study of LLaMA3 quantization and Evaluating Quantized Large Language Models both report this collapse at 2 bits, with GSM8K the most fragile task.

Two pieces of prior work motivate the claim directly. First, Cheng et al. prove that post-hoc renormalization cannot recover the loss surface below 3 bits. Renormalization is an invertible correction, so its failure is exactly what information loss would produce, and the deficit should predict where it fails. Second, the LoRA paradox in the LLaMA3 study shows that fine-tuning a quantized model on Alpaca can make it worse. This is consistent with fine-tuning trying to restore information that is no longer present and overfitting the calibration set instead.

Measuring the Deficit

Let $T_l$ be the full-precision output of layer $l$ on input $X$, and let $T^q_l$ be the same output when the layer is quantized. Let $Z \sim \mathcal{N}(0, \sigma_l^2 I)$ be Gaussian noise added in channel-standardized coordinates. The noise sets a resolution, and standardizing the channels keeps the few massive activation channels from dominating. The deficit is

$$\Delta_l(\sigma_l) = h(T_l + Z \mid T^q_l) - h(Z) = I(T_l + Z;\, X \mid T^q_l),$$

where $h$ is differential entropy. In words, it is the information about the input that the smoothed full-precision activation holds beyond what the quantized activation already holds. The smoothing follows High-Dimensional Smoothed Entropy Estimation via Dimensionality Reduction, which adds isotropic Gaussian noise so that entropies of deterministic network maps are finite. That paper measured continuous networks only. We apply the same smoothing to quantized ones.

Our first departure from prior practice is how we set $\sigma_l$. It is not a free hyperparameter. We choose it as the largest noise level that, when injected at layer $l$ of the full-precision model, raises validation loss by less than 1%. Distortion below the network's own noise tolerance is therefore free by construction.

We estimate the deficit with an upper bound. For any conditional density model $q_\phi$ (a decoder that predicts the full-precision activation from the quantized one),

$$\hat\Delta_l = \mathbb{E}\left[-\log q_\phi(T_l + Z \mid T^q_l)\right] - \tfrac{D}{2}\log(2\pi e \sigma_l^2) \;\ge\; \Delta_l,$$

where $D$ is the width of the layer. The bound holds because cross-entropy upper-bounds conditional entropy. This is the same argument Approximating mutual information of high-dimensional variables using learned representations uses for its cross-prediction regularizer.

We borrow that argument but deliberately do not use that paper's full method. Its low-rank hypothesis fails here, because $T^q_l$ is roughly $T_l$ plus noise, so the dependence between them is full-rank. We also avoid MI lower bounds such as MINE and InfoNCE. Understanding the Limitations of Variational Mutual Information Estimators shows that these saturate and violate additivity as the true MI grows. Hidden states are plentiful (millions of tokens), so the constraint on the estimator is dimension, not sample size.

We use a hierarchy of decoders. The simplest is a linear-Gaussian decoder, which reduces the estimate to a closed-form log-determinant of the whitened residual. Above that are a small MLP mean with low-rank-plus-diagonal covariance and a conditional normalizing flow. Comparing them tells us whether nonlinear recoverability matters at all.

Research Program

The work has three stages.

Diagnosis. We quantize one layer of Llama-2-7B at a time, at 2 bits and near 1 bit, using the AQLM codebase (Vahe1994/AQLM). For each layer we compute the deficit, mean-squared error, and Hessian-weighted error. We then apply a fixed, identical repair budget: continuous scale and codebook tuning only, which is the P step of PV-Tuning, for a fixed token count. We record the loss left over after repair on WikiText-2, C4 and GSM8K.

Localization. We ask whether the reasoning-versus-fluency dissociation corresponds to a small set of high-deficit layers. If the rule is right, those layers should drive GSM8K collapse while leaving perplexity nearly unchanged.

Allocation. We assign mixed codebook sizes per layer at an average budget of about 2 bits, in proportion to each layer's marginal deficit reduction per bit. We then run the same full PV-Tuning pipeline as every other allocation rule.

We also examine one interaction. Incoherence processing, as in QuIP and QuIP#, rotates the weights and undoes the rotation afterward. The layer's activations, and hence the true deficit, are the same in both bases. What changes is how much information survives rounding. We expect incoherence to lower the deficit rather than only the mean-squared error, and we test this directly.

Baselines

The naive reference points are uniform bit allocation followed by the same PV-Tuning budget, and, for the diagnosis stage, ranking layers by plain mean-squared error. If the deficit does not beat mean-squared error, the core claim is false.

The strongest existing method is PV-Tuning with uniform allocation on AQLM codebooks. PV-Tuning reports Llama-2-7B at about 2 bits reaching WikiText-2 perplexity 5.71, against 6.19 for QuIP# and 6.64 for AQLM. It is the Pareto-leading pipeline in the file and ships with public code, so we run it ourselves with the paper's calibration data (RedPajama) and token budget. Every allocation rule receives the same fine-tuning tokens and GPU hours. Beating it means lower WikiText-2 perplexity and higher GSM8K accuracy at an equal average bit count. The file has no reported GSM8K number for 2-bit Llama-2-7B under PV-Tuning (no reported result found on Lacuna for this), so we measure it ourselves.

Two sensitivity-based allocators compete directly on the allocation question. The first is Hessian-trace allocation, the salience signal behind CRVQ's selection of sensitive channels. The second is AlphaQ's calibration-free spectral heavy-tail score, adapted from experts to layers. Both receive the same bit budget and the same repair pipeline.

Three ablations isolate our ingredients. The linear-Gaussian deficit tests whether nonlinear decoding matters. A deficit computed with a fixed $\sigma$ instead of the calibrated $\sigma_l$ tests whether matching the network's noise tolerance matters. A learned-decoder mean-squared residual, with no entropy units or noise calibration, is the closest simple competitor.

Why not just…?

Why not just use mean-squared error or output KL divergence? Both measure distortion, and our hypothesis is precisely that distortion overstates repairable damage. The diagnosis stage settles this. We compute the partial correlation between deficit and post-repair loss after controlling for mean-squared error. If it is near zero, mean-squared error is sufficient and the proposal fails.

Why not just use Hessian sensitivity? Hessian-weighted error is a second-order estimate of how much loss the distortion causes before any repair. It says nothing about invertibility. QuIP shows that Hessian-based rounding alone (OPTQ) collapses Llama 2 70B to perplexity 123.91 at 2 bits, while incoherence processing reaches 6.33. That gap is evidence that the geometry of the error, not only its Hessian-weighted size, decides the outcome. The same allocation experiment tests the question directly.

Why not just fit a decoder and report its regression residual? This is the real threat, because our estimator is itself a decoder. The two differences are the noise-calibrated resolution $\sigma_l$ and the use of log-likelihood units, which make deficits comparable across layers of different width. If the plain residual ablation matches the full deficit, we will report that the information-theoretic framing added interpretation but no predictive power.

Why not just use AlphaQ's spectral scores? They are calibration-free, which is a genuine advantage. But they score static weights and cannot see how quantization error interacts with the activations the model actually produces. The authors also leave generality beyond mixture-of-experts models open. AlphaQ runs as a baseline, and if it ties with our method we will recommend it for its lower cost.

Decisive Checks

The first decisive check is the diagnosis stage on Llama-2-7B and Llama-2-13B, across all layers at two bit-widths. The claim survives only if the deficit's Spearman correlation with post-repair GSM8K accuracy and post-repair WikiText-2 perplexity exceeds that of mean-squared error, Hessian-weighted error, and the residual ablation. We will bootstrap confidence intervals over layers.

The second check is allocation. Deficit-guided allocation must beat uniform PV-Tuning at an average of about 2 bits on WikiText-2, C4, GSM8K, MMLU, and the common-sense suite (PIQA, ARC, HellaSwag, WinoGrande, BoolQ). We expect the largest gains on GSM8K. A perplexity-only gain would weaken the localization claim.

The third check is predictive rather than comparative. The deficit should predict where renormalization fails, the result of Cheng et al.. It should also predict when Alpaca fine-tuning hurts rather than helps, the LoRA paradox. If fine-tuning gains concentrate in low-deficit layers, and losses appear when adapters sit on high-deficit layers, the recoverability rule has explanatory power beyond bit allocation.