FairAdaBN: Mitigating Unfairness with Adaptive Batch Normalization and Its Application to Dermatological Disease Classification

Zikang Xu, Shang Zhao, Quan Quan, Qingsong Yao, S. Kevin Zhou

TL;DR

FairAdaBN is a novel framework that tackles demographic bias in medical AI by making batch normalization layers adaptive to sensitive attributes like skin tone or gender. By specializing normalization parameters for different subgroups while sharing core feature-extraction layers, the method significantly improves model fairness with minimal impact on diagnostic accuracy.

What Problem Does This Paper Address?

Deep learning models in healthcare often suffer from "model unfairness"—a phenomenon where a model performs significantly better for one demographic group (the "privileged" group) than another (the "unprivileged" group). In dermatology, for instance, models trained on predominantly light-skinned populations often underperform when applied to patients with darker skin tones.

Existing solutions usually fall into two camps:

  1. Fairness through unawareness: Ignoring sensitive attributes during training, which often fails because neural networks "leak" demographic info through correlated features.
  2. Independent modeling: Training separate models for each group, which is inefficient and degrades performance because each model sees less data.

The authors seek a middle ground: a single model that shares general knowledge across all groups but maintains specific "adapters" to handle the unique statistical distributions of different demographic subgroups.

Methods

The authors introduce three key components to create a fairness-aware architecture:

  • FairAdaBN (Fair Adaptive Batch Normalization): Standard Batch Normalization (BN) uses a single set of parameters ($\gamma, \beta$) and statistics ($\mu, \sigma$) for the entire dataset. FairAdaBN replaces these with subgroup-specific versions. While the convolutional layers (the "knowledge") are shared across all groups, the normalization layers are parallelized. This allows the model to align feature maps for different subgroups dynamically.
  • Statistical Disparity Loss ($L_{SD}$): To ensure the model doesn't just learn to separate groups but actually treats them fairly, the authors add a novel loss term. This term penalizes the model if the distribution of its predictions differs significantly between subgroups during a mini-batch.
  • FATE Metric (Fairness-Accuracy Trade-off Efficiency): Improving fairness often comes at the cost of overall accuracy. The authors propose FATE to quantify this trade-off, measuring the normalized improvement in fairness relative to the drop in accuracy. A higher FATE score indicates a more efficient "win-win" or a "low-cost" fairness improvement.

Datasets

The researchers evaluated their method on two major dermatological benchmarks:

  • Fitzpatrick-17k: 16,577 images across 9 diagnostic categories. The sensitive attribute is skin tone (categorized into "light" and "dark").
  • ISIC 2019: 25,331 images across 9 categories. The sensitive attribute is gender (male vs. female).

In both cases, the datasets were split into privileged and unprivileged groups based on historical performance disparities (e.g., "dark" skin and "male" were treated as unprivileged in certain contexts based on initial baseline analysis).

Key Insights and Ideas

  • Normalization as a Bias Filter: The core "aha moment" is that bias often manifests as a shift in feature statistics. By isolating these statistics into subgroup-specific BN layers, the model can "standardize" features from different demographics before they reach the next layer of the network.
  • Efficiency through Sharing: Unlike training independent models, FairAdaBN only adds a tiny number of parameters (the affine parameters of BN layers are negligible compared to weights in Convolutions). This allows the model to benefit from the full dataset's diversity while remaining sensitive to subgroup differences.
  • The Pareto Frontier: The authors emphasize that fairness isn't a binary state but a trade-off. Their FATE metric encourages researchers to look for models that sit on the "Pareto frontier"—the point where you get the most fairness for the least accuracy "tax."

Key Results

  • Superior Fairness: FairAdaBN achieved the lowest scores across multiple fairness metrics, including Equalized Odds (EOdd) and Equal Opportunity (EOpp0/1), on both datasets.
  • High FATE Scores: On the Fitzpatrick-17k dataset, FairAdaBN achieved a FATE score of $48.79 \times 10^{-2}$, which is more than double the score of the next best mitigation method (Independent modeling at $22.63 \times 10^{-2}$).
  • Portability: The authors successfully plugged FairAdaBN into several popular backbones (ResNet-152, VGG-19, and DenseNet-121), proving the method is architecture-agnostic.
  • Ablation Success: Removing either the $L_{SD}$ loss or the FairAdaBN layers resulted in a significant drop in fairness, confirming that both the architectural change and the optimization constraint are necessary.

Limitations and Future Work

  • Inference Requirements: A primary limitation is that FairAdaBN requires the sensitive attribute (e.g., the patient's skin tone or gender) to be known at the time of testing/inference to select the correct normalization branch.
  • Accuracy Drop: While FairAdaBN has a better trade-off than other methods, it still results in a slight drop in overall accuracy compared to a "vanilla" (and unfair) model.
  • Scope: The study focused exclusively on dermatology. The authors note that future work should evaluate the framework on other medical imaging tasks like chest X-rays or brain MRIs, as well as different tasks like segmentation and detection.

Summary updated

Abstract

Deep learning is becoming increasingly ubiquitous in medical research and applications while involving sensitive information and even critical diagnosis decisions. Researchers observe a significant performance disparity among subgroups with different demographic attributes, which is called model unfairness, and put lots of effort into carefully designing elegant architectures to address unfairness, which poses heavy training burden, brings poor generalization, and reveals the trade-off between model performance and fairness. To tackle these issues, we propose FairAdaBN by making batch normalization adaptive to sensitive attributes. This simple but effective design can be adapted to several classification backbones that are originally unaware of fairness. Additionally, we derive a novel loss function that restrains statistical parity between subgroups on mini-batches, encouraging the model to converge with considerable fairness. In order to evaluate the trade-off between model performance and fairness, we propose a new metric, named Fairness-Accuracy Trade-off Efficiency (FATE), to compute normalized fairness improvement over accuracy drop. Experiments on two dermatological datasets show that our proposed method outperforms other methods on fairness criteria and FATE. Our code is available at https://github.com/XuZikang/FairAdaBN .
BibTeX
@inproceedings{xu2023fairadabn,
  author = {Zikang Xu and Shang Zhao and Quan Quan and Qingsong Yao and S. Kevin Zhou},
  title = {{FairAdaBN: Mitigating Unfairness with Adaptive Batch Normalization and Its Application to Dermatological Disease Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  url = {https://openreview.net/forum?id=x58x7QoAUu},
}

Loading related content...

Figure