TL;DR
This paper presents the results of the autoPET2 challenge, which tested how well AI models trained on one dataset could segment tumors in PET/CT scans from different sources. The key takeaway is that while models perform well on data similar to their training set, they struggle significantly when faced with variations in institutions, patient populations, pathologies, or even different imaging tracers, highlighting a major hurdle for real-world deployment.
What Problem Does This Paper Address?
Accurate segmentation of cancerous lesions in PET/CT scans is crucial for diagnosis, staging, and treatment monitoring. While deep learning has shown promise, most models are trained and evaluated on data from a single institution or scanner. This raises a critical question: can these models generalize to new, unseen data that reflects the diverse clinical reality of different hospitals, patient demographics, and imaging protocols? This paper addresses this "domain generalization" problem, investigating whether AI models can maintain their performance when applied to data that differs from their training set, a necessary step for widespread clinical adoption.
Methods
The autoPET2 challenge was designed as a comprehensive biomedical segmentation task focused on domain generalization. Seventeen international teams were tasked with training machine learning models on a large, single-source dataset of whole-body 18F-FDG PET/CT scans (1,014 samples). These models were then evaluated on a diverse, multi-institutional test set of 200 samples, encompassing variations in institutions, pathologies, patient populations, and even a different radiotracer (PSMA). Performance was measured using three key metrics: Dice Similarity Coefficient (DSC) for segmentation accuracy, and average false-positive (FPV) and false-negative (FNV) volumes to quantify errors. The challenge also included analyses of ranking stability and qualitative error patterns.
Datasets
- Training Data: 1,014 anonymized whole-body 18F-FDG PET/CT scans from 900 patients, acquired at a single institution (University Hospital Tübingen - UKT). This dataset was also used for the first autoPET challenge.
- Test Data: 200 PET/CT scans from 3 institutions, divided into 5 distinct domains:
- UKT (n=50): In-domain data from the same institution and imaging distribution as the training set.
- UKT-Patho (n=25): Data from the same institution but with different primary tumors.
- LMU (n=25): Data from a different institution (LMU) with different scanners and patient populations.
- UKE-Pediatric (n=50): Data from a different institution (UKE) exclusively from pediatric patients, representing significant age and weight distribution differences.
- LMU-PSMA (n=50): Data from LMU using a different radiotracer (18F-PSMA or 68Ga-PSMA-11), representing a substantial domain shift.
Key Insights and Ideas
- Domain Generalization is a Major Hurdle: The core finding is that AI models trained on a single dataset struggle significantly when applied to data from different sources, even with similar imaging protocols. This "domain shift" is a critical bottleneck for real-world deployment.
- Out-of-Domain Performance Plummets: While models excel on data similar to their training set (in-domain), their performance deteriorates substantially on datasets from different institutions, patient populations, or with different tracers.
- False Positives are a Persistent Problem: A significant challenge across many models is the generation of false positives, often due to physiologic uptake (e.g., muscle activity) or artifacts. Reducing these while maintaining sensitivity is key.
- Pediatric and PSMA Datasets are Particularly Challenging: The UKE-Pediatric and LMU-PSMA datasets proved to be the most difficult for the models, indicating that age-related differences and different radiotracers pose significant generalization challenges.
- nnU-Net Remains a Strong Baseline: The nnU-Net framework was the most popular choice among participants and consistently performed well, highlighting its robustness as a starting point for segmentation tasks. However, even top-performing teams using nnU-Net did not achieve perfect generalization.
- Ensembling Offers Limited Gains: Simple majority-vote ensembles provided only marginal improvements, suggesting that simply combining existing models doesn't fully solve the generalization problem. An "oracle ensemble" (hypothetically selecting the best prediction for each case) showed much higher potential, indicating room for improvement.
Key Results
- Best Performance: The top-performing team achieved an average Dice Similarity Coefficient (DSC) of 0.5038, a mean false-positive volume (FPV) of 87.8388 mL, and a mean false-negative volume (FNV) of 8.4154 mL on the test set. This indicates moderate segmentation quality even for the best models.
- In-Domain vs. Out-of-Domain: Average DSC on the in-domain UKT dataset was high (median 0.7355), but dropped significantly for UKE-Pediatric (median 0.3239) and LMU-PSMA (median 0.1840).
- FPV Issues: The LMU-PSMA dataset showed exceptionally high FPVs, orders of magnitude larger than other domains, significantly impacting overall performance.
- Lesion Detection Sensitivity: Smaller lesions and those with lower SUVmax were less reliably detected across all domains, a known challenge in PET/CT segmentation.
- Ranking Stability: Bootstrap analysis revealed that no single team consistently outperformed all others across different resampled datasets, indicating a close competition and the influence of dataset variations.
Limitations and Future Work
- Limitations:
- The challenge focused on "single-source domain generalization," meaning models were trained on one dataset and tested on others. Future work could explore multi-source training or fine-tuning strategies.
- The evaluation metrics (DSC, FPV, FNV) are coarse and don't fully capture the nuances of segmentation accuracy (e.g., a single voxel error can still count as detected).
- Variability in reference annotations (ground truth) due to multiple expert readers is a known issue and a potential source of domain shift that was not explicitly quantified.
- The study design aggregates the effects of multiple domain shift factors (scanner, population, tracer, annotation), making it difficult to isolate the impact of each individual factor.
- Future Work:
- The authors emphasize the pressing need for more diverse public PET/CT datasets to support robust ML development and improve algorithm robustness.
- Future challenges could aim to disentangle the contributions of individual domain shift factors.
- Exploring more sophisticated ensembling techniques or methods that explicitly handle domain shifts could lead to further performance gains.
- Investigating model calibration and the use of organ priors or multistage approaches for improved segmentation accuracy are suggested.
Summary updated