Shrinking a neural network for a phone or an embedded medical device, by removing channels (pruning) or storing weights in fewer bits (quantization), usually costs only a point or two of overall accuracy. That small cost is not spread evenly. It falls mostly on rare and underrepresented examples, and so on minority demographic groups. The methods that currently protect those groups during compression all need to know each training example's demographic label, such as skin type or sex. In clinical and consumer settings these labels are often missing, noisy, or legally off-limits. This note proposes a way to protect them without any labels. First, find the examples on which the compressed model stops agreeing with the original full-size model. Then use those examples, and only those, to decide which layers keep their channels and bits, and to decide what the compressed model is retrained to preserve.
The core bet is that disagreement between the original network and its compressed version is a usable stand-in for "the minority group is being hurt." Earlier auditing work hints at this, since the examples compression breaks are concentrated in underrepresented subgroups. If the bet holds, a single unlabeled calibration set is enough to steer a joint pruning-and-quantization procedure so that it keeps the capacity those examples depend on. Demographic labels are then needed only to evaluate the result, never to produce it.
Core Claim
Characterising Bias in Compressed Models introduced Compression Identified Exemplars (CIEs), the examples where compressed and uncompressed models diverge. The paper found that they concentrate in minority subgroups. On CelebA at 95% sparsity, the false positive rate rose by 49.54% for the rare Male subgroup and by only 6.32% for Female. At 90% sparsity, CIEs went from 0 to over 3,200 while overall accuracy stayed above 94%. That paper, like its predecessor What Do Compressed Deep Neural Networks Forget?, only audits. It never feeds the signal back into compression.
In the other direction, every method that actively optimizes fairness during compression requires group labels:
- FairQuant learns per-layer bit-widths from minority-group gradients.
- Towards Fairness-aware Adversarial Network Pruning trains a discriminator on protected attributes.
- FaHaNa and BiaslessNAS score candidate architectures with a per-group unfairness penalty.
We claim that disagreement with the original model, measured on held-out unlabeled data, can replace group labels in two places. The first is allocation, meaning how many channels and bits each layer keeps. The second is retraining, meaning which examples the compressed model is pushed hardest to match. The claim is that this recovers most of the worst-group accuracy that label-using methods recover, at the same bit and latency budget.
This is a claim about preserving the original model's fairness, not improving on it. Adversarial pruning reports ending up fairer than the dense network. Our method has no labels and therefore no mechanism to exceed the original, and we do not claim it can.
Research Program
Let $p_T(x)$ be the original model's output probabilities and $p_S(x)$ those of the current compressed model. For each unlabeled calibration example $x_i$, the disagreement is the L1 distance used for Taylor CIE, $d_i = \lVert p_T(x_i) - p_S(x_i) \rVert_1$.
The procedure alternates three steps.
Step 1: rank examples by disagreement. Start from a cheap compressed model, either uniform 4-bit or magnitude channel pruning to the target budget. Compute $d_i$ for every example on a held-out unlabeled split. The split must be held out because CIE is blind on training data when the original model has fit it perfectly, a failure mode noted by Hooker et al.
Step 2: allocate capacity using the most-disagreeing examples. Take the set $H$ of the top $k$ percent of examples by $d_i$. For each layer $l$ and each candidate action, compute a first-order sensitivity score: how much removing a channel group or dropping a bit would increase disagreement on $H$. Summed over the examples in $H$, this is
$$s_l = \sum_{i \in H} \left| \nabla_{\theta_l} \mathrm{KL}\big(p_T(x_i)\,\|\,p_S(x_i)\big)^\top \Delta\theta_l \right|,$$
where $\theta_l$ are layer $l$'s weights and $\Delta\theta_l$ is the weight perturbation the action would cause. We then solve a knapsack problem under a global budget in average bits and measured latency. This is the same first-order Taylor machinery FairQuant uses. The difference is that FairQuant's gradients come from a labeled minority group, and ours come from a set selected purely by disagreement.
Step 3: retrain with disagreement-weighted distillation. Knowledge distillation trains the compressed model to match the original model's output probabilities. Here each example is weighted by
$$w_i \propto \exp(d_i/\tau),$$
where $\tau$ is a temperature that controls how sharply the weighting concentrates on the most-disagreeing examples. We minimize $\sum_i w_i\,\mathrm{KL}(p_T(x_i)\,\|\,p_S(x_i))$. The target is the original model's outputs, not ground-truth labels. This matters because the examples compression breaks are about twice as likely to be mislabeled (20% vs 9%, per Hooker et al. 2019). Fitting labels on them would chase noise, while matching the original model only asks for fidelity.
We iterate these three steps two or three times. Structured channel pruning and mixed precision are decided jointly, which addresses the untested interaction between the two identified in the file.
Analytic prediction. The header/tail finding of FaHaNa says group-divergent computation sits in later layers. If that is right, disagreement-guided allocation should, without being told, spend more bits and channels on the tail. We will check whether it does.
Baselines
Naive reference points. The first is the uncompressed model, which is the ceiling our method aims to preserve. The second is off-the-shelf compression at the same budget: uniform 4-bit post-training quantization with analytical clipping and bias correction as in Post training 4-bit quantization, plus magnitude channel pruning. FairQuant reports how badly uniform 4-bit does on Fitzpatrick17k with ResNet18: average accuracy falls from 50.6% to 23.4%, and TinyViT's worst-group accuracy falls to near zero.
Strongest label-free competitor. This is uniform-weight distillation combined with a generic sensitivity-based mixed-precision allocation computed on random calibration data. It is our method with the two key ingredients removed. It gets the same architecture, budget, calibration data, retraining epochs, and hyperparameter search trials. To beat it, we must show higher worst-group accuracy and a smaller group gap at equal average bits and equal measured latency. The gain must hold across three seeds and not come from lower overall accuracy.
Strongest label-using method (an oracle). This is FairQuant, which reports recovering ResNet18 on Fitzpatrick17k to 45.33% average accuracy at about 4 bits. It also reports TinyViT worst-group accuracy at 4–6 bits comparable to or better than uniform 8-bit. On CelebA, adversarial pruning reports a DEO of 0.2149 at 80% pruning, against 0.5071 for a sparse network trained from scratch. DEO is the difference in equalized odds, the gap between groups in true and false positive rates. Both run with group labels, which we deny our own method. Success means recovering at least most of the gap between uniform compression and FairQuant, for example landing well above 23.4% and close to 45.33% on the same setting. Matching or exceeding FairQuant would be a strong result, but it is not required.
Ablations. Each isolates one ingredient:
- Disagreement used only for allocation.
- Disagreement used only for distillation weighting.
- A randomly chosen set $H$ of the same size, which controls for the effect of simply concentrating attention on any subset.
- A group-oracle $H$ that contains the true minority-group examples. This shows how much our label-free set loses relative to perfect knowledge of who is in the minority.
Why not just…?
Why not just use plain knowledge distillation from the dense model? Distillation is label-free and already pushes disagreement down. It is the most serious threat to this proposal. Our concern is that averaging over the whole dataset lets the few thousand broken examples be swamped by the many examples that already agree. Hooker et al.'s 90%-sparsity numbers, where overall accuracy stays above 94% while CIE accuracy sits near 40–50%, suggest average fidelity says little about tail fidelity. This is still only a hypothesis. The uniform-distillation baseline above settles it directly. If uniform distillation matches our worst-group numbers, the reweighting and the allocation step add nothing, and we would report that.
Why not just predict demographic labels with a classifier and then run FairQuant? This only moves the label problem somewhere else. It requires labeled data to train the attribute predictor. It deploys a sensitive-attribute inference system, which is often exactly what regulation forbids. And it inherits the predictor's errors on the very groups it misrepresents. It also only covers attributes someone thought to name, whereas disagreement is agnostic to which axis is harmed, intersections included. We will include this variant where public attribute labels allow it, as a comparison point, not as a method we recommend.
Why not just adjust decision thresholds per group after compression? Threshold shifting needs group labels at inference time. More fundamentally, it cannot restore signal that is gone. Hooker et al. show compressed models drop to near-random accuracy on CIEs, and Rethinking Bias Mitigation notes that post-processing struggles on hard multi-class problems. Fitzpatrick17k has 114 classes, so this applies directly.
Why not just rebalance or augment the training data? Architecture carries bias of its own even under fixed data. Randomly initialized networks already show demographic disparities (Dooley et al.), and Fair Identity Normalization outperformed oversampling and augmentation. Rebalancing also requires the labels we assume are unavailable.
Proposed Experiments
Benchmarks and models. We use Fitzpatrick17k (light I–III vs dark IV–VI) and ISIC 2019 (sex, skin tone) with ResNet18, ResNet50, DeiT-Tiny, and TinyViT, matching FairQuant's setup. We also use CelebA (Blond Hair across gender and age) with ResNet-18 and MobileNet, matching Hooker et al. and adversarial pruning.
Budgets. Average bit budgets are 6, 4, and 3 bits. Structured sparsity is 50% and 80%.
Metrics. We report overall accuracy, worst-group accuracy, group accuracy gap, and DEO on CelebA. We also report the count of CIEs remaining relative to the original model. For hardware cost, we report model size and measured latency on a Raspberry Pi, following FaHaNa's measurement practice.
Decisive checks. The first check comes before any method is run. We measure how strongly the top-$k$ disagreement set is enriched for minority groups, expressed as the ratio of minority share in $H$ to minority share overall. For reference, Blond Male is 0.85% of CelebA. If enrichment is weak, the premise fails and we stop.
The second check is the random-$H$ versus disagreement-$H$ versus oracle-$H$ comparison. It measures how much of the oracle's benefit the label-free signal captures.
The third check is whether disagreement-guided allocation places capacity in later layers, testing the header/tail account against an independent signal.
The fourth check addresses the Occam-versus-capacity tension. If some pruned configurations end up fairer than the dense model, as adversarial pruning reports, we examine whether our fidelity objective blocks those gains. That would mark the cost of working without labels.
References
- Characterising Bias in Compressed Models
- What Do Compressed Deep Neural Networks Forget?
- FairQuant: Fairness-Aware Mixed-Precision Quantization for Medical Image Classification
- Towards Fairness-aware Adversarial Network Pruning
- The larger the fairer?: small neural networks can achieve fairness for edge devices
- Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Dataset
- Post training 4-bit quantization of convolutional networks for rapid-deployment
- Rethinking Bias Mitigation: Fairer Architectures Make for Fairer Face Recognition
- Equitable artificial intelligence for glaucoma screening with fair identity normalization