Certifying Quantized Models Against Their Full-Precision Parent Without Labels

Teams that ship language and vision models to phones and edge devices usually ship a compressed copy. The model's weights are rounded to 8, 4, or even 2 bits, a process called quantization. Statistical safety wrappers such as conformal prediction are calibrated on held-out labeled data. Conformal prediction, in plain terms, turns a model's scores into answer sets that contain the right answer at least, say, 90% of the time. Those guarantees silently break when the deployed model is the quantized copy and the deployed inputs come from a new domain. The fix that works today is to collect fresh labels for every compressed variant on every new domain, and that is exactly what edge deployments cannot afford.

The observation behind this proposal is that quantization differs from an ordinary distribution shift in one useful way: the "before" model is still available. On unlabeled deployment inputs we can check whether the quantized model's answer set contains the full-precision model's answer set. A simple inequality turns that label-free containment rate into a finite-sample bound on how much coverage quantization can cost. The result is that one labeled calibration of the full-precision model is enough to certify any number of quantized variants (GGUF Q8 through Q2, QuIP 2-bit, INT8 TensorRT builds) on a new domain, using only unlabeled inputs.

Core Claim

Let $C_{fp}(x)$ be the conformal answer set of the full-precision model and $C_q(x)$ the set of a quantized variant, with its own threshold $\lambda_q$. Let $X, Y$ be drawn from the deployment distribution. Whenever the true answer is in the full-precision set and the full-precision set is contained in the quantized set, the true answer is in the quantized set. That gives

$$ P(Y \in C_q(X)) \;\ge\; P(Y \in C_{fp}(X)) \;-\; P\big(C_{fp}(X) \not\subseteq C_q(X)\big). $$

The last term involves no labels. It can be estimated on $n$ unlabeled deployment inputs with a Clopper–Pearson upper bound, and it decreases monotonically in $\lambda_q$. We can therefore pick the smallest $\lambda_q$ whose upper bound falls below a user budget $\varepsilon$ by scanning thresholds in a fixed sequence, so no multiplicity correction is needed. The scan uses the monotone testing idea in Adaptive Learn-then-Test, which develops sequential, anytime-valid threshold selection.

The guarantee is that, with probability $1-\delta$ over the unlabeled sample, the quantized model's deployment coverage is within $\varepsilon$ of the full-precision model's deployment coverage. If the full-precision model carries its own shift-valid guarantee, a union bound transfers it to every quantized variant. That guarantee can come from one round of labeled recalibration on the target domain, done once on a server, or from weighted conformal prediction.

This is relative certification. It makes no claim to repair domain shift for free. It splits a compound shift into a domain part, which costs labels once per model family, and a quantization part, which costs no labels per variant. This decomposition is new. The closest existing work, Conformalized Large Language Models under Configuration Shift, studies quantization only on static test data. It lists compound shift as open, and its only working fix requires $k=50$ labels per configuration.

Research Program

The work has three parts.

Estimator and its open-ended extension. For multiple-choice QA, answer sets are subsets of the options, and containment is checked exactly. For open-ended QA, we follow the sampled-candidate construction that the configuration-shift paper uses on Natural Questions and TriviaQA. Containment is defined over semantically clustered candidates, scored by both models on the same sampled pool, so that the event is still label-free. A selective-answering variant handles systems that return one answer or abstain, in the style of COIN, which bounds the error rate among answered questions using Clopper–Pearson intervals on full-precision models. It replaces containment with "the quantized model answers only when its answer matches the full-precision model's answer or the full-precision model would also abstain."

Compound-shift audit. This addresses the empirical-validation gap directly. We measure whether quantization and domain shift interact super-additively, meaning whether a 2-bit model loses more coverage on a new domain than its in-domain quantization loss plus the full-precision model's domain loss. There are two reasons to expect this. Less Precise Can Be More Reliable found that quantization improves calibration and robustness to synthetic corruptions but hurts semantic and stylistic shifts such as ImageNet-Sketch. Evaluating Quantized Large Language Models found reasoning collapses first below 4 bits. The audit runs on the actual shipped binaries (GGUF kernels, TensorRT INT8) rather than simulated rounding, because the certificate is about the deployed artifact.

Tightness analysis. The bound is loose when the quantized model reorders scores within the full-precision set without changing which answers are correct. We will report the gap between the certified $\varepsilon$ and the realized coverage loss as a function of bit-width. We will relate it to the score-space Kolmogorov–Smirnov distance from the configuration-shift paper. That paper reports this distance is a conservative population-level lower bound (conservativeness ratio 1.00), and its authors say explicitly that it is not a finite-sample certificate. Our quantity is a finite-sample certificate but requires access to the full-precision model.

Baselines

The naive reference is a static threshold calibrated with the full-precision model on source data and applied unchanged to the quantized model on target data. This is the setting in which the configuration-shift paper reports undercoverage rates of 20% for quantization shift alone at a 90% target, against under 3% without shift.

The strongest existing method is labeled recalibration, Recal. It re-estimates the threshold for each quantized variant from $k=50$ labeled target examples, and the same paper reports it reduces undercoverage to 6–8%. We run Recal on identical splits with $k \in \{10, 25, 50, 100\}$ per variant. We report the total label cost across all variants against our method's one-time full-precision label cost. Our method beats Recal if it matches the 6–8% undercoverage level, with comparable average set size, while using labels for only one model instead of one per variant. If Recal with equal total labels dominates on coverage and set size, the method has failed.

The second baseline calibrates each quantized model on its own source calibration data. This removes the "configuration" part of the shift, but domain shift remains.

The third is weighted conformal prediction from Conformal Prediction Under Covariate Shift. It reweights calibration scores by an estimated ratio of deployment to calibration input densities. Here the ratio is estimated by a classifier on frozen embeddings, and the method is applied directly to the quantized model. That paper reports it restores coverage from 82.2% to 91.0% on the low-dimensional UCI Airfoil data. No reported result was found on Lacuna for weighted conformal prediction on LLM or ViT embeddings.

The fourth is temperature scaling with a softmax threshold. Less Precise Can Be More Reliable reports it lowers Expected Calibration Error from 6.9% to 1.1% on quantized VLMs, but it carries no coverage guarantee.

Two ablations isolate the key ingredient, which is using the full-precision sets as a label-free reference. The first replaces containment with matching the quantized model's unlabeled score quantile to the full-precision model's score quantile, with no set-level comparison. The second replaces containment with a worst-case-over-all-labels score gap. All methods share the same unlabeled pool size $n$, the same calibration splits, and 100 random splits per configuration.

Why not just…?

Why not just calibrate each quantized model on its own data? This removes the mismatch between calibrating one model and deploying another, and in-domain it may be all one needs. We include it, and we expect it to tie with our method when there is no domain shift. Under domain shift it inherits the quantized model's own sensitivity, which the VLM evidence suggests is worse on semantic shifts. Restoring validity then requires labeled target data per variant. The compound-shift experiments test exactly this.

Why not just use Recal with 50 labels? It works, and with a single deployed variant and cheap labels it is the right choice. Our claim is about cost: each variant, kernel, and domain multiplies that cost, whereas our method pays once. We also check the configuration-shift paper's "Calibration Paradox," where larger calibration sets lock in a misaligned threshold under persistent shift. That would affect Recal's source-side counterpart, though not Recal on target labels. If equal-budget Recal wins, we will report it.

Why not just reweight by density ratios? Quantization changes the model's predictive mapping. The weighted method's guarantee assumes the label distribution given the input is unchanged, and that assumption covers data shift, not model change. Ratio estimates on high-dimensional embeddings also shrink the effective sample size. We run it on the same embeddings to measure both failure modes.

Why not just control disagreement with the full model, as early-exit methods do? Fast yet Safe already bounds a label-free "consistency risk" between early exits and the full network. Our estimator borrows that spirit. That work calibrates on exchangeable source data and states that it does not handle shift. Its consistency notion is not converted into a coverage bound under a target distribution. Our contribution is that conversion: estimating containment on unlabeled target inputs and pairing it with a shift-valid full-precision guarantee.

Proposed Experiments

The language models are Llama 3/3.2, Gemma 3 and Qwen in GGUF Q8/Q6/Q4/Q2, plus QuIP 2-bit. QuIP reports WikiText2 perplexity of 6.33 at 2 bits on Llama 2 70B, against 123.91 for OPTQ, so 2-bit models are functional enough to test. Labeled full-precision calibration uses MedQA. Deployment shifts are MedMCQA, MMLU-medical, and the register shifts (patient-facing, clinical-note). Investigating Linear Probe Robustness documents these as real shifts, with AUROC drops of 0.21 to MedMCQA, 0.12 to MMLU-medical, and about 0.10 across registers. For general domains we calibrate on MMLU and deploy on CommonsenseQA. Open-ended evaluation uses Natural Questions and TriviaQA.

In vision, we calibrate ViT, ConvNeXt and CLIP-style models on ImageNet and deploy on ImageNet-V2, -R, -Sketch and CIFAR-10-C, under INT8 and INT4 including Rotation+LSQ. This is where the VLM audit predicts quantization helps with corruptions and hurts with semantic shift.

The metrics are coverage, undercoverage rate across splits (defined as in the configuration-shift paper), average set size, selective-answering power at fixed risk, and total labels consumed.

Three results are decisive. First, validity: realized coverage loss must stay at or below the certified $\varepsilon$ in at least a $1-\delta$ fraction of splits at every bit-width. Any systematic violation would reveal non-i.i.d. unlabeled pools or a flaw in the open-ended containment definition. Second, usefulness: at Q4 and above, the certified $\varepsilon$ should be small enough (around 1–3 points) that quantized sets are barely larger than Recal's. At Q2 we expect either honest, large $\varepsilon$ or large sets, and saying "do not deploy this variant" without labels is itself a useful output. Third, the interaction measurement: the size and sign of super-additive coverage loss under compound shift, broken down by shift type. This would settle whether the vision finding that quantization makes models "more reliable" carries over to autoregressive QA, or reverses there.