BiasPruner: Debiased Continual Learning for Medical Image Classification

Nourhan Bayasi, Jamil Fayyad, Alceu Bissoto, Ghassan Hamarneh, Rafeef Garbi

Introduction

Deep learning models in medical imaging face a critical dual challenge: catastrophic forgetting and shortcut learning. When models learn new tasks sequentially, they often overwrite previous knowledge. Simultaneously, they tend to latch onto "spurious correlations"—such as skin tone, patient age, or gender—rather than genuine pathological features. Existing Continual Learning (CL) methods often preserve or even amplify these biases. BiasPruner addresses this by introducing "intentional forgetting," selectively pruning network units that learn these biased shortcuts to create fair, robust subnetworks.

Methodology: Intentional Forgetting

BiasPruner operates on the principle that neural networks are "lazy," fitting easy, biased samples before complex clinical features. The framework follows a three-stage pipeline:

  1. Bias Scoring: The model is initially trained with Generalized Cross Entropy (GCE) loss. Units are scored based on activation differences between "easy" samples (likely biased) and "hard" samples (likely unbiased).
  2. Pruning and Fine-tuning: Units with high bias scores are pruned. The remaining subnetwork is fine-tuned using Weighted Cross Entropy (WCE) to emphasize hard samples, forcing the model to learn clinically relevant features.
  3. Debiased Knowledge Transfer: For sequential tasks, BiasPruner allocates specific, frozen subnetworks for old tasks while utilizing remaining capacity for new ones, preventing both forgetting and bias transfer.
The BiasPruner framework: identifying and pruning biased units to enable fair knowledge transfer across sequential tasks.

For inference, the model employs a task-agnostic "maxoutput" strategy, automatically selecting the subnetwork with the strongest response without requiring task-specific labels.

Experimental Evaluation

The method was evaluated on three benchmarks with known sensitive attributes:

  • FITZ (Fitzpatrick17K): Skin lesion classification (6 tasks) biased by skin tone.
  • HAM (HAM10000): Skin lesion classification (3 tasks) biased by patient age.
  • NIH (ChestX-Ray14): Thorax disease classification (3 tasks) biased by patient gender.

Notably, BiasPruner does not require explicit labels for these sensitive attributes during training, making it highly suitable for privacy-constrained medical environments.

Key Results and Discussion

BiasPruner consistently outperformed state-of-the-art CL methods in both diagnostic accuracy and fairness metrics. On the FITZ dataset, it achieved an overall balanced accuracy of 0.512, significantly higher than standard CL methods like EWC (0.324).

Performance comparison on the FITZ dataset. BiasPruner (Exp D) achieves superior accuracy and fairness (lower EOD) compared to standard CL baselines.

Beyond accuracy, BiasPruner significantly improved fairness. It achieved the lowest Equal Opportunity Difference (EOD) of 0.202 on FITZ, indicating more equitable performance across demographic groups. Probing experiments confirmed that the model's internal features were less predictive of sensitive attributes, verifying that the pruning effectively "unlearned" the bias rather than just hiding it.

Conclusion

BiasPruner demonstrates that forgetting can be a feature rather than a bug in medical AI. By selectively removing biased network components, the framework enables the development of models that are both continually learnable and fundamentally fairer. Future work will explore scaling this pruning logic to larger architectures like Vision Transformers and medical foundation models.

Summary updated

BibTeX
@inproceedings{bayasi2024biaspruner,
  author = {Nourhan Bayasi and Jamil Fayyad and Alceu Bissoto and Ghassan Hamarneh and Rafeef Garbi},
  title = {{BiasPruner: Debiased Continual Learning for Medical Image Classification}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=CqXwhG2Vuz},
}

Loading related content...

Figure