Quantifying the Impact of Population Shift Across Age and Sex for Abdominal Organ Segmentation

Kate Cevora, Ben Glocker, Wenjia Bai

Introduction

Deep learning models for medical image segmentation often encounter a "performance cliff" when transitioning from controlled research environments to real-world clinical settings. This phenomenon, known as domain shift, occurs when the data a model encounters during deployment differs significantly from the data it was trained on. While much research has focused on hardware-induced shifts, this study investigates the often-overlooked impact of population shift—variations in patient demographics such as age and sex—on the segmentation of abdominal organs.

The Three Pillars of Domain Shift

In medical imaging, domain shifts are generally categorized into three types:

  1. Acquisition Shift: Variations stemming from different hardware (e.g., CT vs. MRI) or scanner settings.
  2. Annotation Shift: Inconsistencies in how different radiologists or institutions label anatomical structures.
  3. Population Shift: Differences in the patient populations themselves, including age, sex, ethnicity, and disease prevalence.

While acquisition shift is a well-recognized challenge, population shift remains under-studied in the context of abdominal imaging. If an AI model performs poorly on specific demographic groups—such as women or the elderly—due to underrepresentation in training data, it introduces significant concerns regarding clinical fairness and patient safety.

Methodology: Subgrouping and the $\Delta P$ Metric

To quantify these effects, the researchers utilized nnU-Net, the state-of-the-art framework for medical image segmentation. They conducted experiments using two major public datasets: TotalSegmentator (TS) (1,082 subjects) and AMOS (500 subjects). Crucially, the authors obtained patient-level demographic metadata (age and sex) which is often missing from public releases.

The study employed a rigorous subgrouping strategy:

  • Sex: Male vs. Female.
  • Age: "Under 50" (U50) vs. "Over 70" (O70).

To measure the impact of shifting between these groups, the authors introduced the Performance Gap ($\Delta P$). This metric calculates the change in performance (Dice scores and Hausdorff distance) when a model trained on one subgroup is tested on another, normalized against average performance. This allowed for a direct comparison between population shift and cross-dataset shift (training on one dataset and testing on another), the latter being the traditional benchmark for difficulty in the field.

Results: The Magnitude of Population Shift

The study revealed that population shift is a "first-class" problem in medical AI. The drop in performance caused by age or sex shifts was comparable in magnitude to the drop caused by switching between entirely different datasets. Specifically, cross-dataset shifts resulted in a 5-11% Dice score drop, while population shifts accounted for a 1-6% drop.

Key findings include:

  • Organ Sensitivity: The kidneys were particularly sensitive to population shifts. In the TS dataset, shifting from a male-trained model to female test data resulted in a 6% drop in Dice score and a 95% increase in the 95th percentile Hausdorff Distance (HD95), indicating significant boundary errors.
  • Asymmetric Bias: The impact of shift is not always reciprocal. A model trained on males might struggle significantly with female anatomy, while a model trained on females might generalize better to males, depending on the specific dataset.

The "Diversity Proxy" and the Limits of Demographic Labels

Perhaps the most significant insight from this research is that demographic labels are not a perfect proxy for feature diversity. The authors found that a model trained exclusively on one demographic (e.g., males) sometimes outperformed a model trained on the target demographic (e.g., females).

This occurred because the male subgroup in that specific dataset exhibited higher "morphological diversity"—greater variation in organ shapes, sizes, and positions. To test this, the researchers used the standard deviation of organ volumes as a proxy for image feature diversity. They found a positive correlation between this diversity metric and the model's ability to generalize. This suggests that achieving a 50/50 demographic split in training data is less important than ensuring the training set captures a wide range of anatomical variety.

Conclusion and Future Directions

The study concludes that population shift is a critical factor in the robustness of abdominal segmentation models. To move toward more fair and reliable AI, the authors suggest two primary paths:

  1. Beyond Demographic Labels: Research should shift from focusing solely on labels like age and sex toward developing metrics that quantify image-level diversity, such as shape and texture variation.
  2. Advanced Augmentation: Traditional data augmentation techniques (like rotation or scaling) are insufficient to simulate the complex anatomical changes associated with aging or biological sex. New methods are required to synthetically induce these morphological shifts during training.

By acknowledging and quantifying the impact of population shift, this research provides a foundation for building AI systems that are not only accurate in the lab but also equitable and robust in the clinic.

Summary updated

BibTeX
@inproceedings{cevora2024quantifying,
  author = {Kate Cevora and Ben Glocker and Wenjia Bai},
  title = {{Quantifying the Impact of Population Shift Across Age and Sex for Abdominal Organ Segmentation}},
  booktitle = {FAIMI/EPIMI @ MICCAI},
  year = {2024},
  doi = {10.1007/978-3-031-72787-0\_9},
}
Figure