Calibrating Compressed Reasoning Models to Know When to Stop

When a large language model is compressed after training, either by rounding its weights to a few bits (quantization) or by deleting weights (pruning), the algorithm decides how to compress by running a small "calibration set" of 64 to 256 text samples through the network and copying what it sees. The choice of those few samples matters enormously. Redrawing them with a different random seed can move downstream accuracy by nearly ten points. For reasoning models, which write long internal chains of thought before answering, a second symptom appears: compressed versions think longer than the originals, second-guessing and re-verifying steps they used to commit to. The literature on reasoning length shows that past a problem-dependent point, longer chains are less accurate. Read together, these findings suggest that much of the damage compression does to reasoning models runs through one channel. The compressed model loses its sense of when it is done.

This note proposes to test that claim and act on it. We will measure whether calibration-set choices that make a compressed model overthink are the same choices that make it inaccurate. We will test whether the size of the length shift, which can be measured on unlabeled prompts, predicts which calibration set will work better than perplexity does. Finally, we will build a calibration procedure that targets the moments when the full-precision model decides to stop reasoning and commit to an answer, and gives those moments extra weight.

Core Claim

Our hypothesis is that, in compressed reasoning models, calibration-induced accuracy loss is largely mediated by a distortion of commit behavior. The compressed model reaches the answer the full-precision model would have reached, then fails to stop and talks itself out of it, or drifts. This distortion comes from a small set of activations near the full-precision model's "commit points." Ordinary calibration sets underrepresent these activations, and so do reasoning-trace sets. Even reasoning traces are dominated by mid-chain tokens, and reconstruction objectives weight every token equally. If the hypothesis holds, two things follow. The compressed model's length inflation relative to the original becomes a cheap, label-free diagnostic of calibration quality. And reweighting the calibration objective toward commit points should recover both accuracy and length at no extra inference cost.

What Prior Work Establishes and Leaves Open

The motivating observation comes from Towards Quantization-Aware Training for Ultra-Low-Bit Reasoning LLMs. That paper trains 2-bit Qwen3-8B models with an 80/20 reasoning/pre-training calibration mix and a teacher-guided fine-tuning stage. Along the way it reports "redundant thinking," meaning quantized models generate substantially longer chains because noise lowers their confidence. It treats this as a latency side effect and leaves adaptive mixing and suppression of redundant thinking open. We take the side effect as the object of study and stay in the cheaper post-training regime, with no fine-tuning.

Preserving LLM Capabilities through Calibration Data Curation (COLA) shows that formatting calibration samples as explicit step-by-step reasoning protects reasoning-relevant weights. It selects samples by clustering hidden activations. However, it evaluates non-reasoning base models under fixed generation lengths and never measures trace length. Our method differs from COLA in which tokens inside a sample drive the compression, not in which samples are chosen. Quantization Meets Reasoning finds that early execution errors derail quantized chains into long erroneous continuations, and it repairs them with a small preference-tuning patch. It does not examine the calibration data that produced the damaged weights.

On the reasoning-length side, three papers supply the mechanism we test. When More is Less establishes the inverted-U relation between chain length and accuracy. ThinkBrake shows models often hold the correct answer mid-chain and then overwrite it. Overthinking Reduction with Decoupled Rewards (DECS) defines a "necessary reasoning prefix," the shortest prefix that already determines the answer, using a judge model, and uses it to shape reinforcement-learning rewards. We borrow the prefix idea but locate it without a judge, and we use it to weight a compression objective rather than a training reward.

Two papers act as constraints. On the Impact of Calibration Data in Post-training Quantization and Pruning shows perplexity is blind to downstream swings, and that pruning is far more seed-sensitive than quantization (about 0.2% variance for SpQR). We should therefore expect our effects to be largest for pruning and low-bit settings. Rethinking Pruning Large Language Models shows that aggressive reconstruction overfits calibration activations, and that self-generated calibration samples help. This makes "calibrate uniformly on the model's own traces" a mandatory baseline.

Research Program

All the standard compression methods we target summarize calibration activations through per-token contributions. GPTQ and SparseGPT build a second-moment matrix from activations. AWQ and Wanda use activation magnitudes. That makes a token-weighted objective a drop-in change:

$$\min_{\hat{W}} \; \sum_{t} w_t \, \| W x_t - \hat{W} x_t \|^2 ,$$

Here $x_t$ is the layer input at calibration token $t$, $W$ and $\hat{W}$ are the original and compressed weights, and $w_t \ge 0$ is a per-token weight. Uniform $w_t = 1$ recovers standard practice. For GPTQ and SparseGPT this simply replaces the activation second-moment matrix with its weighted version.

We build calibration samples from the full-precision model's own reasoning traces on a pool of math and science prompts drawn from OpenThoughts-1.2M and the DeepScaleR corpus. On each trace we locate the commit point without a judge, using probing. At regular intervals we append an answer-forcing suffix and record the first position after which the forced answer stably matches the model's final answer. We then set $w_t = 1 + \lambda$ inside a window spanning the last steps before the commit point and the transition to the answer, and $w_t = 1$ elsewhere. Traces are truncated shortly after commit so that post-commit rambling does not dominate the activation statistics. The single new hyperparameter, $\lambda$, and the window width are tuned on a held-out prompt set disjoint from all benchmarks.

The program has three stages. The first stage is a sensitivity atlas. We compress the same model under many calibration choices: source, random seed, and format. For each result we record accuracy, thinking-token count relative to full precision, and a probing-based decomposition of errors. That decomposition separates errors where the compressed model reached the correct intermediate answer and then lost it from errors where it never reached it. The second stage tests whether length inflation on 200 unlabeled prompts predicts downstream accuracy across the atlas. The third stage evaluates commit-weighted calibration against the baselines below.

Baselines

The naive reference point is the uncompressed model and the same model compressed with the default calibration practice: 128 random samples from C4 or WikiText, uniform weights. We run it across five seeds so that its spread, not a single draw, is the bar.

The strongest calibration-curation baseline is COLA (Preserving LLM Capabilities through Calibration Data Curation), using its reasoning-chain formatting and activation clustering. It is the closest published method that targets reasoning preservation through calibration data. Its reported gains are modest but real: HumanEval 64.15 versus 62.10 for WikiText calibration on AWQ 4-bit Qwen2.5-7B, and 1.51 points of math accuracy over random WikiText on SparseGPT-pruned LLaMA3-8B. Alongside it we run the 80/20 OpenThoughts/FineWeb-Edu mix from Towards Quantization-Aware Training for Ultra-Low-Bit Reasoning LLMs in its calibration stage only. That paper reports a 50.45% average improvement over GPTQ and AWQ at 2 bits, but for its full training pipeline, which we do not match in compute. We found no reported result on Lacuna for COLA or the 80/20 mix applied to DeepSeek-R1-Distill models on MATH-500 or AIME, so we will produce those numbers ourselves.

The ablation that isolates our ingredient is uniform-weight calibration on the same truncated self-generated traces, following the self-generated-data finding of Rethinking Pruning Large Language Models. It sees exactly the same tokens and differs only in $w_t$. Two further controls matter. One applies the same total extra weight to a randomly placed window, to rule out a generic effect of reweighting. The other uses untruncated traces with commit weighting, to separate the truncation effect from the weighting effect.

Fairness rules apply to every arm. All arms get the same sample count, the same token budget per sample, the same compression method and bit-width, and the same hyperparameter search budget on the held-out prompt set. Every arm is run on five calibration seeds. We count it as beating the strongest baseline if commit-weighted calibration improves Pass@1 on MATH-500 and AIME over both COLA and the uniform self-trace ablation, by more than their seed spread, at 3-bit GPTQ and 50% SparseGPT. It must also reduce the length ratio to full precision. For reference on what restoration looks like, Quantization Meets Reasoning reports Llama-3.2-3B falling from 37.4% to 31.4% on MATH-500 under 4-bit GPTQ and recovering to 38.2% after its preference-tuning patch. We treat that patch as an orthogonal, training-based upper reference rather than a calibration competitor.

Why not just…?

Why not just fix length at inference time with short-m@k or ThoughtTerminator? This is the most serious threat. Don't Overthink it reports that short-1@k matches majority voting with 40% fewer thinking tokens, and ThoughtTerminator halves R1-1.5B's MATH-500 token spend from over 3000 to around 1500. If running short-m@k on a default-calibrated compressed model recovers as much accuracy as our calibration does, our calibration contribution is redundant, and we will say so. There is reason to expect a gap. The stopping methods rely on the model's own length and confidence signals, and those are exactly the signals compression corrupts. Short-m@k also needs several parallel samples, while calibration costs nothing per query. Stage three settles the question with a crossed comparison of calibration (default, COLA, ours) against decoding (greedy, majority@k, short-m@k) at matched sample counts. If the effects add, both are useful; if they substitute, the cheaper one wins.

Why not just calibrate on reasoning traces, as COLA and the 80/20 mix already do? These methods change which samples are used, but they still weight every token equally. Long traces are dominated by mid-chain and post-commit tokens. The uniform self-trace ablation answers this directly, because it uses our exact data. If it matches commit weighting, the contribution reduces to "use your own truncated traces," which is a smaller but still useful finding.

Why not just use more calibration samples? COLA reports diminishing returns beyond 64 to 128 samples, and Rethinking Pruning Large Language Models shows that more data under uniform reconstruction does not escape overfitting. We nonetheless include a 1,024-sample uniform arm at matched token budget. If commit weighting at 128 samples does not beat it, the token-weighting argument fails.

Why not just validate calibration with perplexity? On the Impact of Calibration Data already shows perplexity misses downstream swings of nearly ten points on RTE for pruned LLaMA-7B. Our stage-two test compares perplexity, layer reconstruction error, and length ratio as predictors on the same atlas, so this claim is measured rather than assumed.

Proposed Experiments

The models are DeepSeek-R1-Distill-Qwen-1.5B and 7B, used in Reasoning Is Not a Race and DECS, and Qwen3-8B, used by Okoshi et al. The compression settings are GPTQ at 4 and 3 bits, AWQ at 4 bits, and SparseGPT and Wanda at 50% sparsity, run through the LLMC toolkit. Evaluation uses MATH-500, AIME 2024/2025, GPQA-Diamond, and LiveCodeBench for accuracy. DUMB500 serves as the overthinking stress test on trivial questions, where any thinking beyond the full-precision length is pure waste. Metrics are Pass@1 and majority@k accuracy, mean thinking tokens, the length ratio to full precision, ThoughtTerminator's overthinking scores on DUMB500, and the standard deviation of each metric across calibration seeds.

The atlas has roughly 4 sources × 5 seeds × 5 methods per model, about 100 compressed models. This makes the mediation analysis feasible.

The first decisive check is mediation. Across atlas models, the accuracy deficit should concentrate on problems where the compressed model's probed intermediate answer was correct and then lost. The deficit should correlate with per-problem length inflation more strongly than with inflation-free errors. If most of the loss is "never reached the answer," commit-point targeting is the wrong lever, and the method stage would be reconsidered.

The second check is prediction. The rank correlation between the unlabeled-prompt length ratio and downstream accuracy should exceed that of perplexity and reconstruction error across the atlas.

The third check is intervention. Commit-weighted calibration should beat COLA and the uniform self-trace ablation on accuracy, length ratio, and seed variance, with the randomly placed window showing no such gain. We expect the largest effects under pruning and 3-bit quantization, where seed sensitivity is highest. Small or absent effects are predicted at 4-bit, and a strong effect at 4-bit would itself be surprising and worth reporting.

The final check is the crossed calibration × decoding grid described above, which determines whether calibration and inference-time stopping are complements or substitutes.