TL;DR
FairQuant is a novel framework designed to ensure that neural network compression doesn't sacrifice algorithmic fairness, particularly in medical imaging. By using "group-aware" importance analysis and learnable bit-widths, it allows models to be compressed to 4-6 bits while maintaining the accuracy and fairness of much larger 8-bit models, preventing performance "collapses" for minority patient groups.
What Problem Does This Paper Address?
Deep learning models for medical image classification (like identifying skin cancer) are often too large to run efficiently on mobile devices or clinical hardware. To fix this, researchers use quantization—reducing the precision of model weights from 32-bit floats to low-precision integers (like 4-bit or 8-bit).
The catch? Standard quantization techniques focus on maintaining average accuracy. In medical settings, datasets are often imbalanced, and a compressed model might still perform well on average while failing miserably for minority subgroups (e.g., patients with darker skin tones). This "fairness gap" is a critical safety issue. The authors address the challenge of compressing models under a strict "bit budget" without leaving vulnerable populations behind.
Methods
The authors introduce FairQuant, which moves away from "one-size-fits-all" quantization in favor of a mixed-precision approach. It consists of three main stages:
- Group-Sensitive Importance Analysis: The framework runs a small "calibration" pass to see which parts of the neural network are most critical for specific subgroups. It uses a first-order Taylor expansion to estimate how much the loss for a specific group would change if a certain weight were removed.
- Budgeted Mixed-Precision Allocation: Instead of giving every layer 4 bits, FairQuant assigns a "bit palette" (e.g., 2, 4, or 8 bits). Layers identified as "important" for fairness or general accuracy get higher precision, while less sensitive units are compressed more aggressively.
- Bit-Aware Quantization (BAQ): This is the most novel component. Rather than picking bit-widths manually, BAQ treats the bit-width as a learnable parameter. During training, the model jointly optimizes its weights and its bit-allocations using a loss function that balances three things: task accuracy, a bitrate budget, and a fairness penalty (minimizing the performance gap between groups).
Datasets
The study evaluates FairQuant on two major dermatology benchmarks:
- Fitzpatrick17k: 16,577 clinical images labeled by 114 diagnostic categories and skin types (I-VI). The researchers grouped these into "lighter" (I-III) and "darker" (IV-VI) skin tones to test for fairness.
- ISIC 2019: 25,331 dermoscopic images where the sensitive attribute is patient sex (male, female, unknown).
They tested these across four different architectures: ResNet18, ResNet50, DeiT-Tiny, and TinyViT.
Key Insights and Ideas
- Mixed-Precision as a Fairness Lever: Not all weights are equally important for all people. By identifying "specialist" units that handle minority group features, we can protect those units with higher precision while still achieving overall compression.
- The 4-bit "Collapse": The researchers found that uniform 4-bit quantization often causes a total performance collapse in vision transformers (like TinyViT), where accuracy for the worst-served group can drop to near zero. FairQuant prevents this "catastrophic forgetting" of minority groups.
- Stability Matters: A key contribution is showing that their learnable bit-allocation (BAQ) is stable across different random seeds and doesn't require "hyperparameter gymnastics" to work effectively.
Key Results
- Recovering Accuracy: On the Fitzpatrick17k dataset using a ResNet18, uniform 4-bit quantization crashed average accuracy from 50.6% to 23.4%. FairQuant (at an average of ~4 bits) recovered this to 45.33%.
- Protecting the Worst-Group: In many cases, FairQuant at 4-6 bits achieved Worst-Group Accuracy comparable to or better than a standard 8-bit uniform model, despite using significantly less memory.
- Efficiency: FairQuant consistently found a more favorable "frontier"—achieving better fairness and accuracy at lower bitrates than standard post-training quantization (PTQ) or quantization-aware training (QAT) baselines.
Limitations and Future Work
The authors acknowledge several areas for improvement:
- Scope of Evaluation: The study is limited to two dermatology datasets and two sensitive attributes (skin type and sex). Medical fairness is intersectional and complex; future work should test more diverse clinical tasks.
- Noisy Labels: Group labels in medical data (like skin type) can be subjective or incomplete, which might affect the importance analysis.
- Fairness Proxies: The fairness loss used during training is a batch-level proxy. The authors suggest that a more "targeted" sweep of fairness regularization weights could reveal even stronger operating points for these models.
Summary updated