TL;DR
This paper investigates how well vendor-provided deep learning models for medical image segmentation perform compared to models custom-trained on institutional data. They found that custom-trained models generally outperform vendor-pretrained ones, especially when accounting for unique institutional practices, and that even a modest amount of local data can yield significant improvements.
What Problem Does This Paper Address?
Accurate segmentation of tumors and organs-at-risk (OARs) in medical imaging is crucial for effective radiation therapy planning. Manual segmentation is time-consuming and prone to variability. While deep learning (DL) offers automated solutions, commercial DL models are often trained on diverse datasets, which may not align with the specific practices and preferences of individual institutions. This can lead to discrepancies in segmentation quality, requiring manual corrections that negate efficiency gains. This study aims to quantify these differences and demonstrate the value of custom-trained models for specific clinical workflows.
Methods
The researchers employed a two-stage 3D U-Net architecture, a common approach in commercial auto-segmentation tools, for both vendor-pretrained and custom-trained models. The process involved a localization stage to identify potential structures, followed by a refined segmentation stage for precise delineation. Custom models were trained using a vendor-supplied DL toolkit and institutional patient data. Performance was evaluated using two key metrics: the Dice Similarity Coefficient (DSC) for overlap accuracy and Mean Surface Distance (MSD) for proximity. Additionally, the impact of segmentation differences on treatment planning was assessed through dosimetric evaluation, comparing mean and maximum dose errors (ADmean and ADmax) to clinically approved contours.
Datasets
The study utilized computed tomography (CT) scans and clinically approved contours from 210 patients across three cancer sites: head-and-neck (HN, 53 patients), breast (left and right, 49+55 patients), and prostate (53 patients). These datasets were split into training, validation, and testing sets. For custom model training, specific splits were used (e.g., 30/5/18 for HN). Vendor-pretrained models were trained on a combination of publicly available datasets and data from vendor-collaborating clinics.
Key Insights and Ideas
- Institutional Practices Matter: The study strongly suggests that the specific ways clinicians contour organs and targets within an institution significantly impact the performance of DL segmentation models. Vendor models, trained on broader datasets, may not capture these nuances.
- Customization is Key for Accuracy: Custom-trained models, leveraging local data, demonstrated superior performance in many cases, particularly for structures where institutional preferences diverge from general guidelines.
- Small Data, Big Impact: A relatively small amount of institutional data (around 30 samples) was found to be sufficient to train custom models that perform comparably to, or even better than, vendor-pretrained models. This democratizes the use of advanced DL tools for institutions with limited data.
- Beyond Geometry: Dosimetric Impact: The differences in segmentation between vendor and custom models translated into measurable differences in radiation dose delivered to critical structures, highlighting the clinical relevance of accurate auto-segmentation.
- Handling Novelties: Custom models showed a better ability to handle specific clinical scenarios, such as the presence of rectal spacers in prostate cancer patients, which vendor models, lacking exposure to such data, struggled with.
Key Results
- Overall Performance: Custom-trained models achieved higher average DSCs and lower average MSDs across all three cancer sites compared to vendor-pretrained models.
- HN: DSC 0.86 vs. 0.81, MSD 0.76 mm vs. 0.81 mm
- Breast: DSC 0.80 vs. 0.67, MSD 2.42 mm vs. 4.85 mm
- Prostate: DSC 0.92 vs. 0.87, MSD 1.21 mm vs. 2.17 mm
- Structure-Specific Improvements: Custom models showed significant improvements for 14 out of 24 OARs.
- Dosimetric Gains: Custom models generally led to smaller errors in mean and maximum dose delivery (ADmean and ADmax) for several critical structures, indicating better alignment with clinical goals.
- Efficiency: The average prediction time for custom-trained models was less than 30 seconds per patient, comparable to vendor models.
Limitations and Future Work
- Absence of Subjective Evaluation: The study lacked subjective evaluation by physicians, which could provide additional insights into clinical acceptability.
- Optimal Data Size: The study did not definitively determine the optimal number of patients required for custom training, though it suggested that a modest amount is sufficient.
- Generalizability: Custom models trained on highly consistent institutional data might underperform in cases with uncommon anatomical variations. Comprehensive quality assurance (QA) is therefore crucial.
- Future Directions: The authors suggest that fine-tuning vendor-pretrained models using transfer learning could be a generalized approach for custom model development, especially given the effort involved in curating large datasets. Further research into the optimal number of training samples and the impact of data quality on model performance is also warranted.
Summary updated