Paying the Robustness Tax Twice: Keeping Minority-Group Accuracy Alive Through Post-Training Quantization

Models that have been deliberately trained not to rely on shortcuts, such as a bird classifier taught to ignore the water in the background, usually give up some average accuracy to do so. This loss is often called the "robustness tax." Separately, practitioners shrink deployed models with quantization, which rounds weights to a few bits. Most quantizers choose their rounding by minimizing average reconstruction error on a small calibration set. We argue these two facts interact badly. The features that let a debiased model handle rare subgroups are carried mostly by rare examples, so an average-error quantizer treats them as noise and rounds them away. The model then pays the robustness tax twice: once in training, and again when compressed, which silently undoes the robustness that was paid for.

The proposal has two parts. The first is a measurement: we test whether debiased models lose more worst-group accuracy under post-training quantization than ordinary models do, and whether that loss is predicted by how little the minority-relevant directions contribute to the quantizer's error objective. The second is a fix that needs no group labels. We reweight the quantizer's calibration objective toward the examples where the full-precision model and a first-pass quantized copy disagree. Our hypothesis is that these examples are disproportionately the minority examples whose features are being destroyed.

Core Claim

Layer-wise post-training quantizers such as OPTQ/GPTQ and the LDLQ rounding used in QuIP choose rounded weights $\hat W$ to minimize $\mathrm{tr}\big((W-\hat W) H (W-\hat W)^\top\big)$. Here $W$ is a layer's weight matrix and $H = \sum_i x_i x_i^\top$ sums the outer products of that layer's inputs $x_i$ over calibration examples $i$. A subgroup that makes up a fraction $p_g$ of the calibration data contributes only about $p_g$ of $H$. Directions used mainly by that subgroup therefore receive little protection, and the protection shrinks as the bit budget tightens.

We replace $H$ with a weighted version:

$$H_w = \sum_i w_i\, x_i x_i^\top, \qquad w_i \propto 1 + \lambda\, \mathrm{KL}\big(p_{\text{full}}(\cdot\mid z_i)\,\|\,p_{\text{quant}}(\cdot\mid z_i)\big)$$

In this expression, $z_i$ is the raw input, $p_{\text{full}}$ and $p_{\text{quant}}$ are the output distributions of the full-precision model and of a first-pass quantized copy, and $\lambda$ sets the strength of the upweighting. The procedure quantizes once with the ordinary $H$, scores disagreement, re-quantizes with $H_w$, and optionally repeats once.

The claim is falsifiable on two counts. First, the disagreement weights should concentrate on true minority groups even though no group labels are used. Second, this label-free reweighting should recover most of the worst-group accuracy that an oracle group-balanced calibration set recovers, without lowering average accuracy below what the robustness tax already costs.

How This Sits Relative to Prior Work

Three papers motivate the gap. Less Precise Can Be More Reliable ran a large evaluation of quantized CLIP-style models. It found that quantization behaves like a low-pass filter on the representation spectrum and that quantized models latch onto backgrounds on Counter-Animal. It also showed that logit tuning, which cuts calibration error from 6.9% to 1.1%, does not repair this. That paper diagnoses the problem but proposes no mitigation, and it never tests models that were debiased on purpose. FairQuant shows the damage directly on dermatology data: uniform 4-bit quantization drops ResNet18 on Fitzpatrick17k from 50.6% to 23.4%, and pushes the worst-served group of TinyViT to near zero. Its remedy is group-labeled Taylor importance plus learnable mixed precision, which requires demographic labels and a training stage. We need neither. Quantization Meets OOD shows that quantization-aware training sharpens the loss landscape and hurts domain generalization, and fixes this with flatness-oriented freezing. It studies style shifts during training, not subgroup shifts under post-training rounding, so it is adjacent to our question rather than overlapping it.

Two lines of work supply the models we compress. Group DRO with strong regularization raises worst-group accuracy from 60.0% to 90.5% on Waterbirds, from 41.1% to 86.7% on CelebA, and from 65.7% to 77.7% on MultiNLI, at a 1–3% average cost. Joint Subspace Estimation removes a spurious subspace while explicitly protecting an orthogonal main-task subspace. Both produce models whose robustness lives in specific, rarely activated directions. Neither has been tested under compression. We use them as the "already paid the tax" starting points.

The quantization machinery comes from existing methods. QuIP's random orthogonal rotations remove weight outliers before rounding. CRVQ protects about 2% of channels selected by a Hessian criterion. We reuse both, changing only the Hessian they see. CRVQ is also the clearest constraint on our design: its channel selection is average-loss driven, and our method predicts that this selection systematically misses channels that matter for minority groups. That prediction is a measurable consequence of the claim.

One finding pushes the other way. The Effect of Model Size on Worst-Group Generalization and QuIP both report that larger pretrained models absorb noise better. A plausible competing hypothesis is that debiased models rely on robust core features and are therefore less fragile under quantization. We treat this as a live outcome rather than an assumption.

Research Program

The first stage is a diagnostic. We take ERM, Group DRO, and JSE-processed versions of the same architecture, quantize each with GPTQ-style rounding, with and without QuIP rotation, at 4, 3, and 2 weight bits, and measure the drop in worst-group accuracy relative to full precision. To test the mechanism, we compute for each layer the fraction of the ordinary Hessian's spectral mass that falls in the subspace minority examples actually use. We estimate that subspace as the top principal directions of minority-group inputs after projecting out majority directions, using held-out group labels for analysis only. We then ask whether this fraction predicts per-layer worst-group damage. For JSE models, we additionally measure whether the angle between the spurious and main-task subspaces drifts after quantization. That drift would mean the removed shortcut has partly come back.

The second stage is the method. We implement disagreement-weighted calibration on top of GPTQ/LDLQ and CRVQ-style channel protection. We check, using held-out group labels only for evaluation, how strongly the weights $w_i$ concentrate on minority groups.

The third stage separates head damage from backbone damage. This tells us whether the problem actually requires changing the rounding at all (see below).

Baselines

The naive reference point is the debiased full-precision model quantized with ordinary GPTQ on a random calibration sample of the same size. This is what a practitioner would do by default. Its worst-group accuracy, compared against the full-precision Group DRO figures above (90.5% Waterbirds, 86.7% CelebA, 77.7% MultiNLI, from the Group DRO paper), defines the gap we try to close. No reported result was found on Lacuna for any post-training-quantized Group DRO or JSE model on these benchmarks, so we establish these numbers ourselves.

The strongest existing competitor is FairQuant. It is the only method in the file that targets subgroup accuracy under quantization, and it has the advantage of group labels and training. Its reported result is 45.33% average accuracy for ResNet18 on Fitzpatrick17k at about 4 bits, against 23.4% for uniform 4-bit and 50.6% at full precision. It is also reported to match 8-bit uniform worst-group accuracy. We run it on Fitzpatrick17k with ResNet18 at its own average bit budget. We beat it only if our label-free method reaches equal or higher worst-group accuracy at the same average bits. We will also report, and not hide, the case where it stays ahead because it uses labels and training.

The oracle is group-balanced calibration: the same quantizer, with a calibration set resampled to equalize the true groups. This is the upper bound for any calibration-only method. We count success as recovering most of the oracle's worst-group gain over the naive baseline without labels.

Two ablations isolate the key ingredient. In the loss-weighted ablation, weights come from the full-precision model's own loss, as in error-set upweighting, rather than from disagreement between the full and quantized models. This tests whether the quantizer's disagreement carries information beyond "hard examples." In the random-weight ablation, weights are drawn at random with the same spread. This tests whether any reweighting helps simply by changing the effective size of the calibration set.

All methods use identical calibration pools, the same number of calibration examples, the same rotation settings, and the same tuning budget for $\lambda$ or its analog, chosen on a validation set that uses worst-group accuracy computed with inferred groups only.

Why not just…?

Why not just balance the calibration set by group? This works when group labels exist. That is exactly why it is our oracle rather than our method. The setting we target is the common one in which labels are absent, which The Effect of Model Size on Worst-Group Generalization flags as the main practical barrier. If disagreement weighting falls far short of the oracle, the honest conclusion is that label-free protection is not yet possible this way, and we will report it as such.

Why not just refit the final linear layer on a small balanced set after quantizing? This is the real threat. If quantization damages only how the head reads out features, a cheap refit fixes everything and changing the rounding is unnecessary. The low-pass-filter finding in Less Precise Can Be More Reliable suggests the damage happens in the backbone's features, which a head refit cannot restore. That is an inference, not a measurement. We settle it directly. We freeze the quantized backbone, train a group-balanced linear probe, and compare its worst-group accuracy with the same probe on full-precision features. If the probe recovers nearly everything, the method half of this proposal collapses into the diagnostic, and we will say so.

Why not just stay at 8 or 4 bits? FairQuant shows subgroup collapse already at uniform 4 bits on dermatology data, but without the rotations that Less Precise Can Be More Reliable found eliminate catastrophic failures. Rotated 4-bit quantization may therefore be safe for worst-group accuracy. The bit sweep reports the exact bit-width at which worst-group loss first appears with and without rotation. If rotated 4-bit is harmless, the contribution narrows to the 2–3 bit regime, which still matters for edge deployment.

Why not just temperature-scale or adjust logits? In Less Precise Can Be More Reliable, logit tuning fixed calibration error but not reliance on backgrounds. Rescaling confidence cannot restore a feature direction that rounding has deleted. We include post-quantization logit adjustment as a cheap extra baseline to confirm this on group benchmarks.

Proposed Experiments

For vision we use Waterbirds and CelebA with ResNet50 and ViT-B, comparing ERM, Group DRO, and JSE variants. For language we use MultiNLI groups and HANS with BERT-base. For the medical setting we use Fitzpatrick17k with ResNet18, for direct comparison with FairQuant. For open-vocabulary models we use Counter-Animal with CLIP-style encoders; Counter-Animal has no reported quantitative number on Lacuna, so we report common-versus-counter accuracy ourselves. Each configuration is evaluated at 4, 3, and 2 weight bits, with and without rotation. Metrics are average accuracy, worst-group accuracy, and the gap between them. We use three seeds per configuration and report confidence intervals.

The decisive checks follow from the claims. The first is whether Group DRO and JSE models lose more worst-group accuracy than ERM at matched bits; either answer is publishable, but only a "yes" motivates the method. The second is whether the per-layer share of Hessian mass in minority directions predicts per-layer damage across layers and models. The third is whether disagreement weights concentrate on true minority groups, measured as the fraction of weight mass on minority groups compared with their base rate. The fourth is whether disagreement weighting beats both the loss-weighted and random-weight ablations and closes most of the gap to the group-balanced oracle. The fifth is whether the linear-probe test shows backbone damage that a head refit cannot repair. A negative result on the fourth check combined with a positive result on the fifth would leave a clean, useful finding: the robustness tax compounds under compression, and only labeled calibration currently prevents it.