BMFT: Achieving Fairness via Bias-Based Weight Masking Fine-Tuning

Yuyang Xue, Junyu Yan, Raman Dutt, Fasih Haider, Jingshuai Liu, Steven McDonagh, Sotirios A. Tsaftaris

In the high-stakes world of medical AI, a model’s "fairness" is more than a metric—it is a critical requirement for equitable healthcare. When dermatology models perform better on light skin tones than dark ones, they risk missing life-threatening diagnoses in under-represented populations. Addressing this "representation bias" typically requires retraining models from scratch or accessing private training data, both of which are often infeasible in post-deployment clinical settings.

A new approach from researchers at the University of Edinburgh introduces BMFT (Bias-based Weight Masking Fine-tuning). This post-processing method "heals" biased models with surgical precision, requiring only a fraction of the time and data previously necessary to achieve robust group fairness.

The BMFT Methodology

The core of BMFT is a two-step "Impair-Repair" strategy guided by a novel masking technique. Rather than retraining the entire network, BMFT identifies and modifies only the specific weights that contribute most to biased predictions.

The BMFT pipeline: (1) Mask Generation identifies high-bias, low-utility weights, followed by (2) a two-step fine-tuning process to debias features and recover classification accuracy.

1. Bias-Importance Masking

Using the Fisher Information Matrix (FIM), the method calculates the sensitivity of every parameter. It targets "high-bias, low-utility" weights—those with a high impact on bias (measured via a differentiable proxy for Equalised Odds) but a low impact on prediction accuracy. These weights are selected for a binary mask $M$.

2. The Impair-Repair Strategy

  • Step 1: Impair (Feature Debiasing): The model fine-tunes only the masked weights in the feature extractor using a small, group-balanced external dataset. This shifts internal representations away from spurious, biased features.
  • Step 2: Repair (Classification Recovery): To ensure diagnostic accuracy is maintained, the final classification layer is reinitialized to zero and fine-tuned. This forces the model to learn how to effectively combine the newly debiased features.

Experimental Evaluation

The researchers evaluated BMFT on melanoma detection across several diverse datasets, including the ISIC Challenge datasets for training and four independent test sets for out-of-distribution (OOD) evaluation: Fitzpatrick-17k, PAD-UFES-20, DDI, and Atlas. These datasets provide critical metadata for skin tone and gender to measure fairness across groups.

Results and Discussion

BMFT consistently outperformed baseline models and existing state-of-the-art (SOTA) methods like FairPrune and LLFT. While pruning methods often destroy accuracy by removing neurons that contain both bias and essential diagnostic information, BMFT’s masking acts as a filter that preserves utility.

Performance comparison of BMFT against SOTA methods across multiple datasets. BMFT shows superior Accuracy (ACC) and Area Under Curve (AUC) while significantly reducing Equalised Odds (EOdds).

Key findings include:

  • Significant Bias Reduction: On the Fitzpatrick-17k dataset, BMFT slashed the Equalised Odds (EOdds)—a primary measure of unfairness—from 0.1745 down to 0.0055.
  • Improved Accuracy: Unlike other debiasing methods that trade off performance for fairness, BMFT improved Accuracy (ACC) from 0.719 to 0.865 on the same dataset.
  • Efficiency: The entire process is remarkably fast, achieving these results in just 5 epochs (roughly 10% of standard training time).
  • OOD Robustness: The fairness gains generalized across completely different datasets, proving the method does not simply overfit to the small tuning set.

Conclusion

BMFT offers a practical, model-agnostic solution for deploying fairer medical AI. By treating bias as an entanglement problem that can be solved through targeted weight masking, it enables clinicians to update pre-trained models for diverse populations without the need for massive compute resources or original training data. Future work will look to extend this success from dermatology to other modalities like X-ray and MRI analysis.

Summary updated

Abstract

Developing models with robust group fairness properties is paramount, particularly in ethically sensitive domains such as medical diagnosis. Recent approaches to achieving fairness in machine learning require a substantial amount of training data and depend on model retraining, which may not be practical in real-world scenarios. To mitigate these challenges, we propose Bias-based Weight Masking Fine-Tuning (BMFT), a novel post-processing method that enhances the fairness of a trained model in significantly fewer epochs without requiring access to the original training data. BMFT produces a mask over model parameters, which efficiently identifies the weights contributing the most towards biased predictions. Furthermore, we propose a two-step debiasing strategy, wherein the feature extractor undergoes initial fine-tuning on the identified bias-influenced weights, succeeded by a fine-tuning phase on a reinitialised classification layer to uphold discriminative performance. Extensive experiments across four dermatological datasets and two sensitive attributes demonstrate that BMFT outperforms existing state-of-the-art (SOTA) techniques in both diagnostic accuracy and fairness metrics. Our findings underscore the efficacy and robustness of BMFT in advancing fairness across various out-of-distribution (OOD) settings. Our code is available at: https://github.com/vios-s/BMFT.
BibTeX
@inproceedings{xue2024bmft,
  author = {Yuyang Xue and Junyu Yan and Raman Dutt and Fasih Haider and Jingshuai Liu and Steven McDonagh and Sotirios A. Tsaftaris},
  title = {{BMFT: Achieving Fairness via Bias-Based Weight Masking Fine-Tuning}},
  booktitle = {FAIMI/EPIMI @ MICCAI},
  year = {2024},
  url = {https://openreview.net/forum?id=khCkgFK67b},
}
Figure