Compressing a language model to 4 or 3 bits usually keeps its accuracy and perplexity close to the original, but it can quietly wreck the model's sense of how sure it is. A question the full-precision model split 51/49 between two answers may come out 95/5 after quantization. Quantization here means rounding the weights to a few discrete levels. The error does not show up in accuracy tables, but it breaks anything built on confidence scores: abstaining when unsure, routing hard queries to a bigger model, verification. The one current remedy for this "uncertainty drift" works at compression time. It hand-picks the small calibration dataset that post-training quantization methods pass through the model to set their rounding ranges, and it biases that set toward examples the model is unsure about. The consequence is that the model's reliability depends on which few hundred sentences were used during compression. Those sentences are often unavailable, private, or mismatched to deployment.
We propose to move the fix into training. A short fine-tuning stage would teach the model to keep its answer-level confidence unchanged when its weights are perturbed the way rounding perturbs them, with most of the effort spent on the examples where confidence is most fragile. If this works, a practitioner can calibrate the full-precision model once with ordinary temperature scaling, then quantize it with any method and any calibration data, including no data at all. The quantized model would inherit the original's calibration without recalibration.
Core Claim
Uncertainty drift under quantization is mostly an example-specific distortion of margins near decision boundaries, not a global sharpening of all predictions. Such distortion can be removed by a training-time consistency objective. That objective penalizes the divergence between the model's answer distribution under its own weights and under simulated rounding noise, weighted toward high-doubt inputs. A model trained this way should show post-quantization calibration error that is low, and also nearly invariant to the choice of quantizer and of quantization calibration set.
The first half of the claim is testable before anything is built, and the project is organized around testing it first. If drift turns out to be a single global temperature shift, a scalar fitted after quantization fixes it, and this proposal is unnecessary. We will report that outcome as a finding.
Relation to Prior Work
The motivating evidence comes from Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models. That paper documents uncertainty drift under GPTQ and AWQ. It counters the drift by scoring candidates with the full-precision model's top-two margin and mixing a fraction of high-doubt examples into the calibration set, a method it calls Doubt-Preserving Quantization (DPQ). On SQuAD2 answerability, DPQ-r75 lowers boundary-accuracy deviation from 0.4553 with WikiText calibration to 0.2125. We reuse its doubt score, but as a training-time weight rather than a data-selection rule. We also target the dependency that DPQ accepts, namely that good uncertainty requires the right quantization data. The paper itself leaves the interaction with post-hoc temperature scaling unresolved, and our diagnostic addresses that question directly.
Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models supplies the closest method and also a tension. It injects noise, scaled by the softmax Jacobian, into pre-attention logits during fine-tuning. This flattens the loss surface and recovers up to 40% relative perplexity and 37% relative ImageNet top-1 under W4A4. It measures only task metrics, however, and its mechanism deliberately diffuses attention. That could make quantized models underconfident rather than well calibrated. Our objective differs in two ways. It anchors the perturbed model to the unperturbed model's own confidence, which penalizes drift in either direction. It also perturbs weights at the scale of actual rounding error rather than perturbing logits.
The in-training calibration literature constrains the design as much as it inspires it. Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration (CALM) trains per-vocabulary logit scaling through a bilevel loop and reports a 41.1% out-of-domain conf-ECE reduction on Llama-3.1. It runs only in full precision and costs 3–4x standard fine-tuning, and it reports that flat entropy penalties hurt accuracy. Differentiable calibration losses such as Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation and Learning Optimal Conformal Classifiers show that calibration targets can be optimized directly. Their gradients rely on large per-sample voxel counts or on splitting mini-batches. The cautionary result is KULeuven at LeQua 2022: Model Calibration in Quantification Learning: trainable kernel-density calibration degraded under strong prevalence shift, while post-hoc Platt scaling won (MRAE 0.117). We take this seriously. Our method does not train calibration itself. It keeps calibration post-hoc and trains only invariance, so the part that can overfit a training distribution stays a single, cheap, refittable scalar.
On the compression side, SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights removes calibration data entirely. It improves Qwen3-1.7B 3-bit WikiText2 perplexity from 32.43 with round-to-nearest to 22.39. Its effect on confidence is untested, and DPQ's results suggest data-free methods may be exactly where drift is worst. TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly attacks the same mismatch between offline calibration data and deployment by recalibrating on test inputs. That moves cost to inference, whereas we move it to a one-time training stage.
Research Program
The first stage is a diagnostic that decides whether the rest is worth doing. We take full-precision models and their quantized versions, varying the quantizer across round-to-nearest, GPTQ, AWQ and SINQ, at 4 and 3 bits. Calibration sets are WikiText, C4, DPQ-r50 and DPQ-r75, plus none for the data-free methods. For each pair we fit the single temperature that best maps quantized answer distributions onto full-precision ones. We then measure what remains after that fit: the residual divergence, and the rate at which the ranking of answer options flips among the top-doubt decile of examples. If a fitted scalar explains most of the drift and leaves few rank flips, the problem is global and post-hoc scaling suffices. If large residuals concentrate on high-doubt examples, which is what DPQ's 51/49 to 95/5 anecdote suggests, a monotone scalar cannot repair them. That outcome justifies a training intervention.
The second stage is the method. During a short fine-tuning run, each step computes the usual task loss. It then adds a consistency term: the model's answer distribution under its current weights $\theta$ versus under weights perturbed by simulated rounding noise $\epsilon$. The objective is
$$\mathcal{L}(\theta) = \mathcal{L}_{\text{task}}(\theta) + \lambda \, \mathbb{E}_{x}\, \mathbb{E}_{\epsilon}\left[ w(x)\, \mathrm{KL}\big(\mathrm{sg}[p_\theta(\cdot \mid x)] \,\|\, p_{\theta+\epsilon}(\cdot \mid x)\big) \right].$$
Here $p_\theta(\cdot \mid x)$ is the model's distribution over answer options (or next tokens) for input $x$, and $\mathrm{sg}$ stops gradients through the unperturbed side so that it acts as a fixed target. The weight $w(x)$ increases as the full-precision top-two margin shrinks, borrowing DPQ's doubt score. The strength of the consistency term is $\lambda$. Each weight entry of $\epsilon$ is drawn uniformly within half a quantization step, where the step size is computed group-wise from the current weights at a bit width sampled from {3, 4}. This mimics rounding error without committing to any one quantizer's grid.
The consistency term needs no labels. It can therefore use unlabeled prompts from task-shaped pools such as the ARC-Challenge and SQuAD2 training splits that DPQ drew from, while the task loss uses Alpaca, as in the JGNI study. One perturbed forward pass per step roughly doubles forward cost, which is still below CALM's reported 3–4x. After training, we fit one temperature on a small full-precision validation split and never refit it after quantization. This is the "calibrate once" condition.
The third stage is analysis. We ask where the invariance comes from by comparing, before and after training, three quantities: attention entropy, logit-outlier magnitude, and the sensitivity of answer margins to per-layer rounding. This tests whether the method works partly through the attention diffusion that JGNI induces, or through a different route such as enlarging margins only where they were fragile.
Baselines
The naive reference is the full-precision model, quantized with WikiText calibration through GPTQ and AWQ and with no data through round-to-nearest and SINQ. It is scored with its full-precision temperature carried over unchanged. This is how most people deploy today, and it defines how large the drift is.
The strongest compression-time competitor is DPQ (Target-Aware Calibration Data Selection), using both the r50 and r75 mixtures, applied to the same base model with the same quantizer settings and candidate pools. On SQuAD2 its reported boundary-accuracy deviation is 0.2125, compared with 0.4553 for WikiText calibration. We count it as beaten if our model quantized with plain WikiText, or with no data through SINQ, matches or goes below DPQ's deviation. The test is equally whether our method stacks with DPQ to go lower still. For calibration error on ARC, MMLU and the other multiple-choice sets, no reported result was found on Lacuna for quantized-model ECE. We will therefore rerun DPQ ourselves and report its numbers alongside ours.
The strongest post-hoc competitor is temperature scaling refit on the quantized model, using a validation split drawn from in-domain data. This gives it an advantage our "calibrate once" condition does not have. Beating it in-domain is not required. The target is matching it in-domain and beating it on out-of-domain task families, where CALM's 41.1% result suggests fitted scalars transfer poorly.
The strongest training-time competitor is JGNI (Jacobian-guided Noise Injection). It gets the same data, the same number of steps, and a hyperparameter sweep of the same size as ours, with a tuning budget matched in GPU hours. CALM, run in full precision and then quantized, is a secondary baseline that tells us whether better full-precision calibration alone carries through compression.
Two ablations isolate the key ingredients. The first replaces the KL consistency term with plain task loss under the same weight noise. This amounts to noise-injection training without the confidence anchor and tests whether anchoring matters or noise alone does the work. The second sets $w(x)$ to a constant, which tests whether doubt weighting matters.
Why not just…?
Why not just refit temperature scaling after quantizing? This is the most serious threat, and the diagnostic stage exists to settle it. If a single fitted scalar leaves little residual divergence and few rank flips among high-doubt examples, we will conclude that the simple fix suffices. We will report that and stop. Even if it does suffice in-domain, refitting requires labeled validation data for every quantized variant, which is exactly the data dependency the compression literature is trying to remove. CALM's out-of-domain results also suggest a fitted temperature transfers badly. The out-of-domain evaluation below measures this directly for quantized models.
Why not just use doubt-aware calibration data, as DPQ does? DPQ works, but it needs task-aligned candidate pools and a new selection for each deployment domain. It also cannot help data-free quantizers like SINQ, which by construction take no calibration data. If our model quantized with SINQ matches DPQ-quantized baselines on boundary deviation and ECE, the data dependency is gone. If it does not, running DPQ on top of our model tells us whether the two remedies are complementary. Neither outcome requires us to argue that DPQ is wrong.
Why not just do quantization-aware training for the target format? Training directly against one quantizer's rounding grid is a real threat for that format, and it may beat us there. Its weakness is that it binds the model to one bit width and one quantizer. The question here is robustness across quantizers and calibration sets. We include a quantization-aware training baseline at 4-bit round-to-nearest under matched compute. We then evaluate it on 3-bit, GPTQ, AWQ and SINQ. The method we propose should win on the spread across settings, even if it loses on the single setting the baseline trained for.
Why not just use JGNI, which already makes models robust to quantization? JGNI protects perplexity and accuracy, but its mechanism flattens attention. That could leave quantized models systematically underconfident, and nobody has measured their ECE. Running JGNI with calibration metrics is itself a contribution. The noise-without-anchor ablation separates "robust to noise" from "confidence stays put," which is the distinction this proposal rests on.
Proposed Experiments and Decisive Checks
Base models are Llama-3.2-3B and Qwen2.5-3B, matching the JGNI setup, plus Qwen3-1.7B, matching SINQ. Evaluation uses option-scored multiple-choice sets: ARC-Challenge, ARC-Easy, MMLU, HellaSwag, PIQA, WinoGrande, OpenBookQA, BoolQ, CommonsenseQA and TruthfulQA. SQuAD2 answerability serves as the boundary-sensitive task. MedMCQA is held out entirely from the consistency-training prompts and serves as an out-of-domain family. Metrics are accuracy, ECE, Brier score, and selective-prediction area under the risk–coverage curve. We also report DPQ's boundary-accuracy deviation, the rate of option-rank flips between full-precision and quantized models among high-doubt examples, and WikiText2 perplexity as a check that nothing broke.
Every model is crossed with every combination of quantizer, bit width and calibration set described above, with W4A4 as a secondary stress test.
The decisive check is a single number per model: the spread of post-quantization ECE across all quantizer and calibration-set combinations, under the frozen full-precision temperature. The central claim holds if our model's spread and mean are both clearly below those of the naive, DPQ, JGNI and quantization-aware training baselines. This must come without an accuracy or perplexity cost larger than the noise across seeds, and it must include SINQ and round-to-nearest, where no calibration data exists. It fails if refit temperature scaling closes the gap, or if the noise-only ablation performs just as well. The first failure would mean the problem is global. The second would mean the confidence anchor adds nothing beyond JGNI-style noise. Either result is informative, and the diagnostic stage should predict which one we see before the main runs begin.
References
- Target-Aware Calibration Data Selection for Preserving Uncertainty in Quantized Language Models
- Jacobian-guided Noise Injection for Quantization Robustness in Large Language Models
- Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibration
- Average Calibration Error: A Differentiable Loss for Improved Reliability in Image Segmentation
- Learning Optimal Conformal Classifiers
- KULeuven at LeQua 2022: Model Calibration in Quantification Learning
- SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights
- TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly