Shortcut learning leads to sex bias in deep learning models for photoacoustic tomography

Marcel Knopp, Christoph J. Bender, Niklas Holzwarth, Yi Li, Julius Kempf, Milenko Caranovic, Ferdinand Knieling, Werner Lang, Ulrich Rother, Alexander Seitel, Lena Maier-Hein, Kris K. DreherInt. J. Comput. Assist. Radiol. Surg 2025 · DOI

Introduction: The Challenge of Shortcut Learning in Clinical AI

As artificial intelligence becomes increasingly integrated into clinical workflows, "shortcut learning" has emerged as a significant barrier to fairness and reliability. Shortcut learning occurs when a deep learning model relies on spurious correlations—features associated with a target label but not causally linked to it—to make predictions. While this phenomenon has been documented in established modalities like X-ray and MRI, its impact on Photoacoustic Tomography (PAT) remains largely unexplored.

PAT is an emerging "light-in, sound-out" imaging modality that combines the high contrast of optical imaging with the deep resolution of ultrasound. This study investigates shortcut learning in the context of Peripheral Artery Disease (PAD), a condition where narrowed arteries reduce blood flow to the limbs. Because PAD prevalence and expression are influenced by biological sex, the researchers sought to determine whether AI models learn to diagnose the disease directly or inadvertently rely on the patient's sex as a diagnostic shortcut.

Causal graph illustrating desired vs. shortcut prediction pathways in PAD diagnosis.

Methodology

The researchers utilized a Convolutional Neural Network (CNN) architecture (EfficientNetV2_B0) to perform two primary tasks: sex classification and PAD diagnosis. To isolate the effects of demographic bias, they constructed training datasets with varying sex-specific prevalence ratios (PR). For instance, a PR of 1 represents a perfectly balanced dataset, while a PR of $\infty$ indicates a dataset where only one sex presented with the disease.

Beyond standard accuracy metrics, the team employed Principal Component Analysis (PCA) and Wasserstein distances to visualize feature encoding. They also utilized transfer learning by freezing the feature extractor of a pre-trained sex classifier and retraining only the output head for PAD diagnosis. This approach allowed them to quantify the overlap between sex-related and disease-related features.

Datasets

The study involved 147 individuals:

  • Primary Dataset: 2D PAT images of the calf muscle captured at three wavelengths (760, 800, and 850 nm) using a clinical MSOT Acuity Echo system.
  • Out-of-Distribution (OOD) Dataset: 525 PAT images from 30 healthy volunteers across different body sites (calf, forearm, and neck) to test the generalizability of the sex classifier.

Key Insights

Sex is "Visible" in PAT Data

A striking finding of the study is that CNNs can classify a patient's sex from calf muscle PAT images with an AUROC of 0.75. This suggests that biological differences—such as skin thickness or fat distribution—leave a distinct "fingerprint" in PAT data that AI can detect, even if it is imperceptible to human radiologists.

The Shortcut Trap and Shared Representations

When training data was imbalanced (e.g., higher disease prevalence in one sex), models shifted their strategy to use sex as a proxy for disease. Transfer learning experiments confirmed this: retraining a sex classifier's head for PAD diagnosis resulted in only a minor performance loss (0.05 AUROC), indicating that the underlying feature representations for sex and PAD are deeply intertwined.

Key Results

The model's ability to identify sex was robust, achieving an AUROC of 0.75 on the calf and generalizing well to other body parts, reaching up to 0.81 AUROC on OOD calf data.

AUROC performance for sex classification across in-distribution and out-of-distribution datasets.

However, this capability leads to a significant performance drop in diagnostic tasks when prevalence shifts occur. Models trained on extremely imbalanced data (PR = $\infty$) dropped to 0.69 AUROC when tested on a balanced population, compared to 0.79 AUROC for models trained on balanced data. This disparity in performance directly translates to increased underdiagnosis in the underrepresented sex.

Discussion and Future Directions

While the study is limited by its sample size and the potential for other confounding factors like BMI or age, it highlights a critical risk in medical AI. The authors deliberately avoided debiasing techniques to provide an "unfiltered view" of how these models fail.

Future work should explore explicitly incorporating sex as an auxiliary feature in a controlled manner, rather than allowing models to infer it implicitly. Furthermore, larger, multi-center studies are essential to ensure that demographic-specific prevalence imbalances do not bake systemic unfairness into clinical AI tools.

Summary updated

BibTeX
@article{knopp2025shortcut,
  author = {Marcel Knopp and Christoph J. Bender and Niklas Holzwarth and Yi Li and Julius Kempf and Milenko Caranovic and Ferdinand Knieling and Werner Lang and Ulrich Rother and Alexander Seitel and Lena Maier-Hein and Kris K. Dreher},
  title = {{Shortcut learning leads to sex bias in deep learning models for photoacoustic tomography}},
  journal = {Int. J. Comput. Assist. Radiol. Surg.},
  year = {2025},
  doi = {10.1007/s11548-025-03370-9},
}
Figure