Quantizing Debiased Models Without Group Labels

Models that have been corrected to stop relying on spurious cues, such as a bird classifier that looks at the background instead of the bird, usually carry that correction in a thin layer of fragile structure. When such a model is compressed for a phone or embedded device by quantization, which stores weights and activations at 4 or 8 bits instead of 32-bit floats, the damage lands disproportionately on the rare examples the correction was meant to protect. Average accuracy barely moves while the worst-served group collapses. On the Fitzpatrick17k dermatology benchmark, uniform 4-bit quantization of a ResNet-18 dropped average accuracy from 50.6% to 23.4%, and on TinyViT it pushed worst-group accuracy to near zero (FairQuant). The existing remedy is mixed precision, meaning a different bit-width for each layer, chosen with gradients computed on group-labeled calibration data. On-device calibration sets almost never carry demographic or spurious-attribute labels.

We propose to allocate bits so that quantization error does not move the decision of the debiased classifier on the examples that classifier finds hardest. The procedure is concrete. First, debias the model by retraining only its last layer, using pseudo-minority examples found without labels in the style of RAD. Then, for each layer and each candidate bit-width, measure how far quantizing that layer pushes the final features along the retrained head's weight direction on those pseudo-minority examples. Finally, solve a small budgeted assignment problem over these scores, and retrain the head once more on the quantized features. The bet is that minority-group failures under quantization come from a few layers whose rounding error happens to line up with the direction the debiased head depends on. If so, a label-free, head-aligned score should find those layers where average-loss sensitivity cannot.

Core Claim

At an edge budget of roughly 4 bits on average, the worst-group accuracy that a last-layer-debiased model loses under quantization can be recovered, without group annotations, by scoring layers on head-aligned error over pseudo-minority samples. This recovery should approach what group-labeled mixed precision achieves, and it should cost almost nothing in in-distribution average accuracy. The claim is falsifiable in two ways. It fails if simply retraining the head after uniform quantization closes the gap. It also fails if ordinary feature-reconstruction sensitivity picks the same layers.

Why This Is Different From What Exists

The closest work is FairQuant. It scores layer importance with first-order Taylor expansions of group-specific losses and learns bit-widths jointly with a fairness penalty. We differ in three respects. We need no group labels. Our sensitivity target is the robust decision direction rather than a group loss. And we treat debiasing and quantization as a pipeline whose order matters, not as a single joint objective.

Layer-wise Adversarial-aware Quantization also allocates bits per layer without calibration data, using Lipschitz bounds. It lifted robust accuracy of a 3-bit ResNet-32 on CIFAR-10 from 20–33% to about 81%. However, it targets norm-bounded adversarial noise and quantizes weights only. Its input-independent bounds cannot tell a minority-relevant direction from any other direction.

The domain-generalization quantization papers, QT-DoG and GAQAT, show that quantization noise can act as a helpful regularizer under domain shift, but both keep bit-widths uniform across layers. Their results motivate the idea that precision interacts with robustness. They do not supply a per-layer allocation rule.

The method components are reused from elsewhere. RAD supplies label-free pseudo-minority discovery. It trains a strongly ℓ1-regularized linear probe that latches onto the shortcut, treats the probe's errors as minority examples, and held about 83.5% CelebA worst-group accuracy under label noise, whereas label-reliant methods fell below 79%. Deep feature reweighting supplies the last-layer retraining step.

Two papers motivate the prediction that damage concentrates in particular layers. Tsoy & Konstantinov show that in full-precision networks, shallow blocks encode spurious features while deep blocks discard core features. H2T-DFR shows that robust signal leaks away before the last layer; it raised CelebA worst-group accuracy from 85.99% to 88.59% by mining intermediate layers. H2T-DFR also constrains our design. Its fix concatenates activations from many layers, which inflates peak activation memory on edge chips. We therefore deliberately keep a single-layer head and spend the budget on precision instead.

Sagawa et al. supply the mechanistic hypothesis. Overparameterized models fit minority examples by memorizing them along low-margin directions. Directions with small margin are exactly the ones a small, aligned perturbation can flip.

Research Program

Let $\phi(x)$ denote the full-precision penultimate features, and let $w$ be the weight direction of the head retrained on RAD pseudo-minority upweighted data. For a two-class task this is one vector; for more classes it is the difference of class rows. Let $\phi_{l,b}(x)$ denote the features when only layer $l$ is quantized to $b$ bits, with activations quantized at the same bit-width. Let $\mathcal{M}$ be the set of pseudo-minority calibration samples, and let $m(x)$ be the full-precision margin of $x$ under the head. The layer score is

$$S_l(b) = \frac{1}{|\mathcal{M}|}\sum_{x\in\mathcal{M}} \left(\frac{w^\top(\phi_{l,b}(x)-\phi(x))}{m(x)}\right)^2 .$$

This is the squared fraction of each hard example's margin that the rounding error of layer $l$ consumes. We choose bit-widths $b_l$ from the palette {2, 4, 8} to minimize $\sum_l S_l(b_l)$, subject to a total weight-memory budget and a peak-activation-memory budget. Treating layer contributions as additive makes this a small knapsack problem. The additivity assumption will be checked directly by comparing summed single-layer scores with the measured score of the jointly quantized model.

After allocation, we freeze the quantized backbone and retrain the head once on quantized features using the same pseudo-minority weights. We then quantize the head to 8 bits, following the boundary-layer evidence that heads are fragile (Rastegari et al.).

The analysis half of the program uses the allocation as an instrument. We will record which layers the score protects across ResNet-18/50 and DeiT-Tiny on Waterbirds and CelebA. We will then test whether those layers coincide with the "core forgetting" blocks or the "spurious encoding" blocks identified by Tsoy & Konstantinov. This answers the open question in the research file of whether low precision in shallow layers filters shortcuts or starves downstream core features.

Baselines

The naive reference point is uniform post-training quantization at 4 bits, and separately at 8 bits, applied to the debiased model with the head left as trained in full precision. This is what an edge toolchain does by default. Its reported failure mode is the FairQuant collapse from 50.6% to 23.4% average accuracy on Fitzpatrick17k with ResNet-18.

The first strong competitor is quantize-then-reweight. We apply uniform 4-bit quantization and then retrain the RAD-style head on the quantized features. This baseline has the same label requirements and nearly the same cost as our method, and it isolates whether per-layer allocation adds anything beyond re-fitting the head.

The strongest existing method is FairQuant run with true group labels. Its reported result is 45.33% average accuracy at about 4-bit average on Fitzpatrick17k with ResNet-18, and worst-group accuracy comparable to 8-bit on TinyViT. We will also run it with RAD pseudo-labels in place of true labels. For fairness, all methods use the same backbones, the same calibration set size, the same bit palette, the same average-bit and activation-memory budgets, and a tuning budget matched in GPU hours. FairQuant's learnable bit-aware training is allowed equal or greater compute than our one-shot allocation. We also reproduce FairQuant's Fitzpatrick17k setting to check that our reimplementation reaches its reported number before we compare against it.

Two ablations isolate the key ingredient. The first replaces $S_l$ with plain feature mean-squared error over all calibration samples, the in-distribution reconstruction criterion typical of layer sensitivity work such as LoaQ. The second keeps head alignment but averages over all samples instead of pseudo-minority ones. Beating the baselines means two things at the same budget. Our worst-group accuracy must exceed quantize-then-reweight and both ablations by a margin larger than the seed-to-seed standard deviation over five seeds. And it must reach at least FairQuant-with-pseudo-labels, and come within a few points of FairQuant with true labels, with average accuracy within one point of the full-precision debiased model.

Why not just…?

Why not just retrain the last layer after uniform quantization? This is the most serious threat. If quantization error is mostly an isotropic perturbation of the features, a re-fitted head will absorb it, and allocation is pointless. The evidence against that is indirect. H2T-DFR shows that minority-relevant signal is already thin by the final layer, and destroyed signal cannot be recovered by re-fitting the head. Our quantize-then-reweight baseline tests this directly. If it matches our method at 4 bits, we will report that as the finding and restrict the claim to the 2–3 bit regime, or drop the claim if it holds there too.

Why not just run FairQuant with RAD pseudo-labels? This is plausible, and we run it. The two differ in what they preserve. FairQuant's Taylor score measures change in group loss. On pseudo-minority sets, that loss is dominated by examples that were misclassified all along, and those contribute large gradients unrelated to quantization. Our margin-normalized, head-aligned score focuses on correctly classified minority examples near the boundary. The head-to-head comparison at matched budget settles which choice matters.

Why not just quantize everything to 8 bits? Uniform 8-bit is often nearly lossless, and it is the right answer when memory allows. The proposal targets the roughly 4-bit regime, where uniform quantization is documented to collapse minority groups (FairQuant). We report the full accuracy-versus-model-size curve so that readers can see where the uniform 8-bit point sits relative to ours.

Why not just subsample the majority group? Sagawa et al. show that subsampling rescues worst-group error, bringing it to 15.1% on CelebA. It needs clean group labels, discards 96.6% of the data, and compresses nothing. It is complementary: a subsampled model still has to be quantized. We include one run that quantizes a subsampled model with our allocation, to check whether the two combine.

Proposed Experiments

The main benchmarks are Waterbirds and CelebA with ResNet-18, ResNet-50 and DeiT-Tiny, reporting worst-group accuracy, average accuracy, model size and peak activation memory. Fitzpatrick17k with ResNet-18 serves as the anchor for direct comparison with FairQuant's reported numbers. On CelebA the full-precision references are DFR at 85.99% worst-group accuracy (H2T-DFR) and RAD-UW at about 83.5% (RAD). No reported result was found on Lacuna for quantized last-layer-debiased models on CelebA or Waterbirds, and no exact Waterbirds DFR number is quoted in the file, so we establish those references ourselves. We will also test robustness to label noise by corrupting 5–20% of the labels in the FairQuant calibration set, mirroring RAD's protocol. The prediction is that FairQuant degrades while our method is unaffected, because it never reads group labels. The deployment check runs the chosen mixed-precision models under an integer runtime on one mobile CPU and reports latency.

Three results would decide the claim. The first is whether quantize-then-reweight is beaten at 4 and 3 bits. The second is whether the head-aligned score beats plain feature mean-squared error, which tests alignment as the mechanism. The third is whether the layers the method protects match the deep "core forgetting" blocks of Tsoy & Konstantinov. A match would link quantization sensitivity to shortcut mechanics. A mismatch would indicate that quantization damage follows a different geometry from the one that governs shortcut learning.