AutoPET Challenge on Fully Automated Lesion Segmentation in Oncologic PET/CT Imaging, Part 2: Domain Generalization

Jakob Dexl, Sergios Gatidis, Marcel Früh, Katharina Jeblick, Andreas Mittermeier, Anna Theresa Stüber, Balthasar Schachtner, Johanna Topalis, Matthias P. Fabritius, Sijing Gu, Gowtham Krishnan Murugesan, Jeff VanOss, Jin Ye, Junjun He, Anissa Alloula, Bartłomiej W. Papież, Zacharia Mesbah, Romain Modzelewski, Matthias Hadlich, Zdravko Marinov, Rainer Stiefelhagen, Fabian Isensee, Klaus H. Maier-Hein, Adrián Galdrán, Konstantin Nikolaou, Christian la Fougère, Moon Young Kim, Nico Kallenberg, Jens Kleesiek, Ken Herrmann, Rudolf A. Werner, Michael Ingrisch, Clemens C. Cyran, Thomas KüstnerJournal of Nuclear Medicine 2025 · DOI

TL;DR

This paper presents the results of the autoPET2 challenge, which tested how well AI models trained on one dataset could segment tumors in PET/CT scans from different sources. The key takeaway is that while models perform well on data similar to their training set, they struggle significantly when faced with variations in institutions, patient populations, pathologies, or even different imaging tracers, highlighting a major hurdle for real-world deployment.

What Problem Does This Paper Address?

Accurate segmentation of cancerous lesions in PET/CT scans is crucial for diagnosis, staging, and treatment monitoring. While deep learning has shown promise, most models are trained and evaluated on data from a single institution or scanner. This raises a critical question: can these models generalize to new, unseen data that reflects the diverse clinical reality of different hospitals, patient demographics, and imaging protocols? This paper addresses this "domain generalization" problem, investigating whether AI models can maintain their performance when applied to data that differs from their training set, a necessary step for widespread clinical adoption.

Methods

The autoPET2 challenge was designed as a comprehensive biomedical segmentation task focused on domain generalization. Seventeen international teams were tasked with training machine learning models on a large, single-source dataset of whole-body 18F-FDG PET/CT scans (1,014 samples). These models were then evaluated on a diverse, multi-institutional test set of 200 samples, encompassing variations in institutions, pathologies, patient populations, and even a different radiotracer (PSMA). Performance was measured using three key metrics: Dice Similarity Coefficient (DSC) for segmentation accuracy, and average false-positive (FPV) and false-negative (FNV) volumes to quantify errors. The challenge also included analyses of ranking stability and qualitative error patterns.

Datasets

  • Training Data: 1,014 anonymized whole-body 18F-FDG PET/CT scans from 900 patients, acquired at a single institution (University Hospital Tübingen - UKT). This dataset was also used for the first autoPET challenge.
  • Test Data: 200 PET/CT scans from 3 institutions, divided into 5 distinct domains:
  • UKT (n=50): In-domain data from the same institution and imaging distribution as the training set.
  • UKT-Patho (n=25): Data from the same institution but with different primary tumors.
  • LMU (n=25): Data from a different institution (LMU) with different scanners and patient populations.
  • UKE-Pediatric (n=50): Data from a different institution (UKE) exclusively from pediatric patients, representing significant age and weight distribution differences.
  • LMU-PSMA (n=50): Data from LMU using a different radiotracer (18F-PSMA or 68Ga-PSMA-11), representing a substantial domain shift.

Key Insights and Ideas

  • Domain Generalization is a Major Hurdle: The core finding is that AI models trained on a single dataset struggle significantly when applied to data from different sources, even with similar imaging protocols. This "domain shift" is a critical bottleneck for real-world deployment.
  • Out-of-Domain Performance Plummets: While models excel on data similar to their training set (in-domain), their performance deteriorates substantially on datasets from different institutions, patient populations, or with different tracers.
  • False Positives are a Persistent Problem: A significant challenge across many models is the generation of false positives, often due to physiologic uptake (e.g., muscle activity) or artifacts. Reducing these while maintaining sensitivity is key.
  • Pediatric and PSMA Datasets are Particularly Challenging: The UKE-Pediatric and LMU-PSMA datasets proved to be the most difficult for the models, indicating that age-related differences and different radiotracers pose significant generalization challenges.
  • nnU-Net Remains a Strong Baseline: The nnU-Net framework was the most popular choice among participants and consistently performed well, highlighting its robustness as a starting point for segmentation tasks. However, even top-performing teams using nnU-Net did not achieve perfect generalization.
  • Ensembling Offers Limited Gains: Simple majority-vote ensembles provided only marginal improvements, suggesting that simply combining existing models doesn't fully solve the generalization problem. An "oracle ensemble" (hypothetically selecting the best prediction for each case) showed much higher potential, indicating room for improvement.

Key Results

  • Best Performance: The top-performing team achieved an average Dice Similarity Coefficient (DSC) of 0.5038, a mean false-positive volume (FPV) of 87.8388 mL, and a mean false-negative volume (FNV) of 8.4154 mL on the test set. This indicates moderate segmentation quality even for the best models.
  • In-Domain vs. Out-of-Domain: Average DSC on the in-domain UKT dataset was high (median 0.7355), but dropped significantly for UKE-Pediatric (median 0.3239) and LMU-PSMA (median 0.1840).
  • FPV Issues: The LMU-PSMA dataset showed exceptionally high FPVs, orders of magnitude larger than other domains, significantly impacting overall performance.
  • Lesion Detection Sensitivity: Smaller lesions and those with lower SUVmax were less reliably detected across all domains, a known challenge in PET/CT segmentation.
  • Ranking Stability: Bootstrap analysis revealed that no single team consistently outperformed all others across different resampled datasets, indicating a close competition and the influence of dataset variations.

Limitations and Future Work

  • Limitations:
    • The challenge focused on "single-source domain generalization," meaning models were trained on one dataset and tested on others. Future work could explore multi-source training or fine-tuning strategies.
  • The evaluation metrics (DSC, FPV, FNV) are coarse and don't fully capture the nuances of segmentation accuracy (e.g., a single voxel error can still count as detected).
  • Variability in reference annotations (ground truth) due to multiple expert readers is a known issue and a potential source of domain shift that was not explicitly quantified.
  • The study design aggregates the effects of multiple domain shift factors (scanner, population, tracer, annotation), making it difficult to isolate the impact of each individual factor.
  • Future Work:
    • The authors emphasize the pressing need for more diverse public PET/CT datasets to support robust ML development and improve algorithm robustness.
  • Future challenges could aim to disentangle the contributions of individual domain shift factors.
  • Exploring more sophisticated ensembling techniques or methods that explicitly handle domain shifts could lead to further performance gains.
  • Investigating model calibration and the use of organ priors or multistage approaches for improved segmentation accuracy are suggested.

Summary updated

Abstract

This article reports the results of the second iteration of the autoPET challenge on automated lesion segmentation in whole-body PET/CT, held in conjunction with the 26th International Conference on Medical Image Computing and Computer Assisted Intervention in 2023. In contrast to the first autoPET challenge, which served as a proof of concept, this study investigates whether machine learning-based segmentation models trained on data from a single source can maintain performance across clinically relevant variations in PET/CT data, reflecting the demands of real-world deployment. <b>Methods:</b> A comprehensive biomedical segmentation challenge on PET/CT domain generalization was designed and conducted. Participants were tasked to train machine learning models on annotated whole-body <sup>18</sup>F-FDG data (<i>n</i> = 1,014). These models were then evaluated on a test set of 200 samples from 5 clinically relevant domains, including variations in institutions, pathologies, and populations and a different tracer. Performance was measured in terms of average dice similarity coefficient, average false-positive volume, and average false-negative volume. The best-performing teams were awarded in 3 categories. Furthermore, a detailed analysis was conducted after the challenge, examining results across domains and unique instances, along with a ranking analysis. <b>Results:</b> Generalization from a single-source domain remains a significant challenge. Seventeen international teams successfully participated in the challenge. The best-performing team reached an average dice similarity coefficient of 0.5038, a mean false-positive volume of 87.8388 mL, and a mean false-negative volume of 8.4154 mL on the test set. nnU-Net was the most commonly used framework, with most participants using a 3-dimensional U-Net. Despite competitive in-domain results, out-of-domain performance deteriorated substantially, particularly on pediatric and prostate-specific membrane antigen data. Detailed error analysis revealed frequent false-positives due to physiologic uptake and decreased sensitivity in detecting small or low-uptake lesions. A majority-vote ensemble offered minimal performance gains, whereas an oracle ensemble indicates hypothetical gains. Ranking analysis showed no single team consistently outperformed all others across ranking schemes. <b>Conclusion:</b> The second autoPET challenge provides a comprehensive evaluation of the current state of automated PET/CT tumor segmentation, highlighting both progress and persistent challenges of single-source domain generalization and the need for diverse public datasets to enhance algorithm robustness.
BibTeX
@article{dexl2025autopet,
  author = {Jakob Dexl and Sergios Gatidis and Marcel Früh and Katharina Jeblick and Andreas Mittermeier and Anna Theresa Stüber and Balthasar Schachtner and Johanna Topalis and Matthias P. Fabritius and Sijing Gu and Gowtham Krishnan Murugesan and Jeff VanOss and Jin Ye and Junjun He and Anissa Alloula and Bartłomiej W. Papież and Zacharia Mesbah and Romain Modzelewski and Matthias Hadlich and Zdravko Marinov and Rainer Stiefelhagen and Fabian Isensee and Klaus H. Maier-Hein and Adrián Galdrán and Konstantin Nikolaou and Christian la Fougère and Moon Young Kim and Nico Kallenberg and Jens Kleesiek and Ken Herrmann and Rudolf A. Werner and Michael Ingrisch and Clemens C. Cyran and Thomas Küstner},
  title = {{AutoPET Challenge on Fully Automated Lesion Segmentation in Oncologic PET/CT Imaging, Part 2: Domain Generalization}},
  journal = {Journal of Nuclear Medicine},
  year = {2025},
  doi = {10.2967/jnumed.125.270260},
}
Figure