FairQuant: Fairness-Aware Mixed-Precision Quantization for Medical Image Classification

Thomas Woergaard, Raghavendra Selvan

TL;DR

FairQuant is a novel framework designed to ensure that neural network compression doesn't sacrifice algorithmic fairness, particularly in medical imaging. By using "group-aware" importance analysis and learnable bit-widths, it allows models to be compressed to 4-6 bits while maintaining the accuracy and fairness of much larger 8-bit models, preventing performance "collapses" for minority patient groups.

What Problem Does This Paper Address?

Deep learning models for medical image classification (like identifying skin cancer) are often too large to run efficiently on mobile devices or clinical hardware. To fix this, researchers use quantization—reducing the precision of model weights from 32-bit floats to low-precision integers (like 4-bit or 8-bit).

The catch? Standard quantization techniques focus on maintaining average accuracy. In medical settings, datasets are often imbalanced, and a compressed model might still perform well on average while failing miserably for minority subgroups (e.g., patients with darker skin tones). This "fairness gap" is a critical safety issue. The authors address the challenge of compressing models under a strict "bit budget" without leaving vulnerable populations behind.

Methods

The authors introduce FairQuant, which moves away from "one-size-fits-all" quantization in favor of a mixed-precision approach. It consists of three main stages:

  1. Group-Sensitive Importance Analysis: The framework runs a small "calibration" pass to see which parts of the neural network are most critical for specific subgroups. It uses a first-order Taylor expansion to estimate how much the loss for a specific group would change if a certain weight were removed.
  2. Budgeted Mixed-Precision Allocation: Instead of giving every layer 4 bits, FairQuant assigns a "bit palette" (e.g., 2, 4, or 8 bits). Layers identified as "important" for fairness or general accuracy get higher precision, while less sensitive units are compressed more aggressively.
  3. Bit-Aware Quantization (BAQ): This is the most novel component. Rather than picking bit-widths manually, BAQ treats the bit-width as a learnable parameter. During training, the model jointly optimizes its weights and its bit-allocations using a loss function that balances three things: task accuracy, a bitrate budget, and a fairness penalty (minimizing the performance gap between groups).

Datasets

The study evaluates FairQuant on two major dermatology benchmarks:

  • Fitzpatrick17k: 16,577 clinical images labeled by 114 diagnostic categories and skin types (I-VI). The researchers grouped these into "lighter" (I-III) and "darker" (IV-VI) skin tones to test for fairness.
  • ISIC 2019: 25,331 dermoscopic images where the sensitive attribute is patient sex (male, female, unknown).

They tested these across four different architectures: ResNet18, ResNet50, DeiT-Tiny, and TinyViT.

Key Insights and Ideas

  • Mixed-Precision as a Fairness Lever: Not all weights are equally important for all people. By identifying "specialist" units that handle minority group features, we can protect those units with higher precision while still achieving overall compression.
  • The 4-bit "Collapse": The researchers found that uniform 4-bit quantization often causes a total performance collapse in vision transformers (like TinyViT), where accuracy for the worst-served group can drop to near zero. FairQuant prevents this "catastrophic forgetting" of minority groups.
  • Stability Matters: A key contribution is showing that their learnable bit-allocation (BAQ) is stable across different random seeds and doesn't require "hyperparameter gymnastics" to work effectively.

Key Results

  • Recovering Accuracy: On the Fitzpatrick17k dataset using a ResNet18, uniform 4-bit quantization crashed average accuracy from 50.6% to 23.4%. FairQuant (at an average of ~4 bits) recovered this to 45.33%.
  • Protecting the Worst-Group: In many cases, FairQuant at 4-6 bits achieved Worst-Group Accuracy comparable to or better than a standard 8-bit uniform model, despite using significantly less memory.
  • Efficiency: FairQuant consistently found a more favorable "frontier"—achieving better fairness and accuracy at lower bitrates than standard post-training quantization (PTQ) or quantization-aware training (QAT) baselines.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Scope of Evaluation: The study is limited to two dermatology datasets and two sensitive attributes (skin type and sex). Medical fairness is intersectional and complex; future work should test more diverse clinical tasks.
  • Noisy Labels: Group labels in medical data (like skin type) can be subjective or incomplete, which might affect the importance analysis.
  • Fairness Proxies: The fairness loss used during training is a batch-level proxy. The authors suggest that a more "targeted" sweep of fairness regularization weights could reveal even stronger operating points for these models.

Summary updated

Abstract

Compressing neural networks by quantizing model parameters offers useful trade-off between performance and efficiency. Methods like quantization-aware training and post-training quantization strive to maintain the downstream performance of compressed models compared to the full precision models. However, these techniques do not explicitly consider the impact on algorithmic fairness. In this work, we study fairness-aware mixed-precision quantization schemes for medical image classification under explicit bit budgets. We introduce FairQuant, a framework that combines group-aware importance analysis, budgeted mixed-precision allocation, and a learnable Bit-Aware Quantization (BAQ) mode that jointly optimizes weights and per-unit bit allocations under bitrate and fairness regularization. We evaluate the method on Fitzpatrick17k and ISIC2019 across ResNet18/50, DeiT-Tiny, and TinyViT. Results show that FairQuant configurations with average precision near 4-6 bits recover much of the Uniform 8-bit accuracy while improving worst-group performance relative to Uniform 4- and 8-bit baselines, with comparable fairness metrics under shared budgets.
BibTeX
@misc{woergaard2026fairquant,
  author = {Thomas Woergaard and Raghavendra Selvan},
  title = {{FairQuant: Fairness-Aware Mixed-Precision Quantization for Medical Image Classification}},
  year = {2026},
  eprint = {2602.23192},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.23192},
}
Figure