Measuring Quantization Error Where the Model Actually Goes

When a large language model is compressed to 2 or 3 bits per weight, it usually keeps writing fluent sentences while losing the ability to carry out multi-step reasoning. Existing compression methods are tuned by minimizing one number. That number is either the error in each layer's output or the model's next-token loss on a fixed text corpus, and in both cases it is computed on text the model did not write. At generation time, though, the compressed model conditions on its own earlier tokens. A small error early in a chain of reasoning moves it onto prefixes the calibration text never covered, and errors compound from there. We propose keeping quantization a single-objective problem but changing where the objective is measured. The new scalar is the divergence between the full-precision and quantized models' next-token distributions, evaluated on prefixes that the quantized model generates itself. We then optimize this objective with the discrete-continuous machinery that already works at 1–2 bits.

The obvious move at this intersection is to replace single-objective bit allocation with multi-objective Pareto search, trading reasoning against safety against memory. We think that move may be premature. Before adding objectives, one should check whether the reasoning and safety collapses reported in the literature come from optimizing the right quantity on the wrong distribution. If they do, a single, better-placed scalar should recover them jointly. That would be a cheaper and more transferable fix, and it can be tested directly.

Core Claim

The accuracy cliff at extreme bit-widths is partly a distribution-shift artifact of how quantized models are calibrated. Calibration loss is measured under teacher forcing, meaning the model is always fed the correct preceding text. Evaluation, by contrast, is autoregressive: the model feeds on its own outputs. Minimizing the following closed-loop objective should recover multi-step reasoning accuracy that teacher-forced objectives miss, at matched compute and bit budget:

$$\mathcal{L}_{\text{on}}(\theta_q) = \mathbb{E}_{x \sim \mathcal{D},\; y \sim p_{\theta_q}(\cdot \mid x)} \sum_{t=1}^{T} \mathrm{KL}\big(p_{\text{fp}}(\cdot \mid x, y_{<t}) \,\|\, p_{\theta_q}(\cdot \mid x, y_{<t})\big)$$

Here $\theta_q$ are the quantized model's parameters (discrete codes plus continuous scales), $\mathcal{D}$ is a prompt pool, $y$ is a response sampled from the quantized model and treated as fixed when taking gradients, $T$ is its length, and $p_{\text{fp}}$ is the full-precision model. The intuition comes from the standard imitation-learning argument. If a model is trained only on the reference's own trajectories, a per-step error $\epsilon$ can compound to $O(T^2\epsilon)$ divergence over $T$ steps. Training on the learner's own trajectories reduces this to $O(T\epsilon)$. Long chain-of-thought is exactly the regime where $T$ is large.

The motivating evidence comes from four papers. ParoQuant, which uses sparse Givens rotations to tame outlier channels, attributes reasoning loss to per-token errors accumulating over long generations. It addresses this by making each token's error smaller, not by changing what is optimized. Evaluating Quantized Large Language Models, a broad empirical study, finds that mathematical reasoning is the most fragile ability. It also observes that dialogue degenerates in a fixed order, from repeated sentences to repeated tokens to gibberish. That order is what one would expect from a model drifting off its training distribution. Quantization Meets Reasoning shows that quantized models keep the format of a solution while failing its intermediate steps. It also shows that preference tuning on 545 curated failure cases recovers MATH-500 accuracy. This suggests the damage is repairable, and repairable from the model's own mistakes.

Research Program

The method reuses an existing optimizer. PV-Tuning shows that straight-through gradient estimates are unstable at 1–2 bits. It replaces them with an alternation between two steps. In the P step, continuous scales and codebooks are updated by gradient descent. In the V step, discrete codes are updated only on a small subset of coordinates with the largest gradients. PV-Tuning drives this alternation with calibration cross-entropy on RedPajama text. We keep the alternation and swap the loss for $\mathcal{L}_{\text{on}}$.

Each outer round runs three stages. First, it samples responses from the current quantized model on a prompt pool that mixes math problems, general instructions, and safety-relevant requests. Second, it scores every prefix with both models. Third, it runs P and V steps on the resulting token-level KL. Because the full-precision model supplies a full distribution at every position, no reward model, preference labels, or curated failure set is needed. The prompts can be unlabeled.

We also study a hybrid that interpolates between teacher-forced and on-policy prefixes with a mixing weight $\lambda$. It starts near teacher forcing, where the badly broken initial quantized model produces useless samples, and anneals toward on-policy. The schedule for $\lambda$ is the only new hyperparameter.

A second, cheaper use of the same idea targets bit allocation. Mixture Compressor assigns 1–3 bits to Mixture-of-Experts modules by solving a linear program over a reconstruction-and-routing score. AlphaQ does the same without calibration data, using the heavy-tailedness of weight spectra. Both are single-objective allocators with a proxy objective. We would measure each block's sensitivity as the increase in $\mathcal{L}_{\text{on}}$ when that block alone is dropped one bit, and feed this score into the same allocator. This isolates whether the sensitivity signal, rather than the allocation algorithm, is what fails on reasoning.

The analysis part of the program asks whether a separate safety or reasoning objective is needed at all. Alignment-Aware Quantization shows that W4A4 reconstruction-based quantization erodes refusals. It repairs this with a contrastive loss that pulls the quantized model toward the aligned model and pushes it away from the unaligned base, with safety and utility summed into one loss. If closed-loop KL to the aligned full-precision model preserves refusals on its own, the push term and the weighting become unnecessary. If it does not, the result is direct evidence that safety needs its own axis.

Baselines

The naive reference points are uniform round-to-nearest quantization and GPTQ-style layer-wise reconstruction at the target bit-width. The file documents how badly these fail below 3 bits. QuIP, which applies random orthogonal rotations before rounding, reports Llama 2 70B WikiText-2 perplexity of 123.91 for 2-bit OPTQ against 6.33 for QuIP. Quantization Meets Reasoning reports Llama-3.2-3B at 31.4% on MATH-500 under GPTQ against a 37.4% full-precision baseline.

The strongest existing competitor on our question is PV-Tuning with its original teacher-forced calibration loss. It shares our optimizer, so any difference is attributable to the objective. It also reports the best 2-bit perplexity in the file: 5.71 WikiText-2 perplexity on Llama 2 7B at about 2 bits, against 6.19 for QuIP# and 6.64 for AQLM. We start from the same AQLM-initialized checkpoints and run both arms with the same code. The two arms get the same number of optimizer steps and the same token budget. For the on-policy arm, that budget counts sampling and teacher-scoring tokens, so it does not receive extra compute. Beating PV-Tuning means higher MATH-500 and GSM8K accuracy at equal bits and equal budget, with WikiText-2 perplexity no more than 0.1 worse. No reported result was found on Lacuna for PV-Tuning on MATH-500 or GSM8K, so we measure both arms ourselves.

The strongest reasoning-repair competitor is the DPO "medicine" procedure from Quantization Meets Reasoning, reported at 38.2% MATH-500 on Llama-3.2-3B. We rerun it on our checkpoints with its 545-pair recipe. Because that recipe uses a small curated set, we compare at its reported compute and also at ours.

For safety, the competitor is AAQ at W4A4 on LLaMA3.1-8B. It reports a SafetyBench score of 60.1, against 57.5 for OSTQuant and 36.6 for GPTQ. Beating it means matching 60.1 without a push term or an unaligned reference model.

For long reasoning at 4 bits, the reference is ParoQuant's 70.1% MMLU-Pro on Qwen3-4B, against 71.0% at FP16 and 68.2% for AWQ. Here our method can be applied on top of ParoQuant's rotations.

The key ablation is teacher-forced KL on full-precision-generated responses. It uses the same prompts, the same teacher distributions, and the same optimizer, and differs only in whose samples form the prefixes. This isolates the on-policy ingredient from the mere use of reasoning-domain data and full-distribution targets.

Why not just…?

Why not just calibrate on reasoning data instead of web text? This is the most serious threat, and it may account for much of any gain. Mixture Compressor's reasoning drop at 2.05 bits, and PV-Tuning's reliance on RedPajama, are both consistent with a simple domain mismatch. Our key ablation answers the question directly. Teacher-forced KL on full-precision chains of thought over the same math prompts is exactly "calibrate on reasoning data." If it matches the on-policy arm, the contribution collapses to a data choice, and we will report that plainly. The prediction that distinguishes the two is that the gap should grow with response length, so we stratify results by solution length.

Why not just do multi-objective Pareto search over reasoning, safety, and size? NEMO did this for CNN mixed precision, using evolutionary search guided by a bandit over accuracy, size, and bit-operations. Nothing prevents porting it to LLMs. But Pareto search only returns trade-offs among the objectives it is handed. If each objective is a teacher-forced proxy, the front is drawn in the wrong space. We therefore run a small scalarization sweep that trades reasoning KL against safety KL. If a single closed-loop objective lands on or near that front, the extra objectives buy little. If the front stays wide, with reasoning and safety genuinely in conflict, multi-objective search is warranted and our result says so.

Why not just fine-tune with preference data on the model's failures, as in the medicine approach? That approach is already partly on-policy, since its pairs come from the quantized model's mistakes. It also works: it reports 38.2% on MATH-500 in minutes. It needs a curated, labeled failure set, however, and it gives a sparse sequence-level signal. We give a dense token-level signal from unlabeled prompts. The matched comparison shows whether density and label-freedom matter. We also check transfer to domains the medicine set does not cover, namely HumanEval and SafetyBench.

Why not just reduce per-token error with better rotations? QuIP and ParoQuant do exactly that, and they are complementary to our method rather than rivals. We apply our objective on top of both. If rotations alone remove the length-dependent gap, our method adds nothing at 4 bits. We expect it to matter mainly at 2 bits and below.

Proposed Experiments

Models are Llama 2 7B at about 2 and about 1 bit, Llama-3.2-3B, Qwen3-4B, and LLaMA3.1-8B at W4A4. This choice lets every cited number be reproduced on its home model. Metrics are WikiText-2 perplexity as a guardrail; MATH-500, GSM8K, and HumanEval accuracy for multi-step ability; MMLU-Pro; SafetyBench; and the zero-shot suite (PIQA, ARC, HellaSwag, BoolQ).

The first decisive check is diagnostic and needs no training. Across roughly forty quantized configurations, varying method, bit-width, and bit allocation, we compute rank correlations with MATH-500 accuracy for three quantities: teacher-forced perplexity, teacher-forced KL, and closed-loop KL. If closed-loop KL does not predict reasoning accuracy clearly better than the other two, the premise fails, and we stop before building the optimizer.

The second check is the matched on-policy versus teacher-forced comparison inside PV-Tuning, with gains broken down by solution length.

The third check replaces the sensitivity scores in the Mixture Compressor and AlphaQ allocators with closed-loop sensitivities on Mixtral at 2.05–2.54 bits. This tests whether allocation fails because of its signal rather than its solver.

The fourth check is the safety test: whether closed-loop KL to the aligned model reaches AAQ's 60.1 on SafetyBench without contrastive terms.

A negative result on any of these is informative. In particular, a persistent reasoning-versus-safety conflict under the closed-loop objective would be the first clean evidence that LLM quantization needs a genuinely multi-objective formulation.