Comparison of Vendor-Pretrained and Custom-Trained Deep Learning Segmentation Models for Head-and-Neck, Breast, and Prostate Cancers

Xinru Chen, Yao Zhao, Hana Baroudi, Mohammad D. El Basha, Aji Daniel, Skylar Gay, Cenji Yu, He Wang, Jack Phan, Seungtaek Choi, Chelain R. Goodman, Xiaodong Zhang, Joshua S. Niedzielski, Sanjay Shete, Laurence E. Court, Zhongxing Liao, Fredrik Löfman, Peter Balter, Jinzhong YangDiagnostics 2024 · DOI

TL;DR

This paper investigates how well vendor-provided deep learning models for medical image segmentation perform compared to models custom-trained on institutional data. They found that custom-trained models generally outperform vendor-pretrained ones, especially when accounting for unique institutional practices, and that even a modest amount of local data can yield significant improvements.

What Problem Does This Paper Address?

Accurate segmentation of tumors and organs-at-risk (OARs) in medical imaging is crucial for effective radiation therapy planning. Manual segmentation is time-consuming and prone to variability. While deep learning (DL) offers automated solutions, commercial DL models are often trained on diverse datasets, which may not align with the specific practices and preferences of individual institutions. This can lead to discrepancies in segmentation quality, requiring manual corrections that negate efficiency gains. This study aims to quantify these differences and demonstrate the value of custom-trained models for specific clinical workflows.

Methods

The researchers employed a two-stage 3D U-Net architecture, a common approach in commercial auto-segmentation tools, for both vendor-pretrained and custom-trained models. The process involved a localization stage to identify potential structures, followed by a refined segmentation stage for precise delineation. Custom models were trained using a vendor-supplied DL toolkit and institutional patient data. Performance was evaluated using two key metrics: the Dice Similarity Coefficient (DSC) for overlap accuracy and Mean Surface Distance (MSD) for proximity. Additionally, the impact of segmentation differences on treatment planning was assessed through dosimetric evaluation, comparing mean and maximum dose errors (ADmean and ADmax) to clinically approved contours.

Datasets

The study utilized computed tomography (CT) scans and clinically approved contours from 210 patients across three cancer sites: head-and-neck (HN, 53 patients), breast (left and right, 49+55 patients), and prostate (53 patients). These datasets were split into training, validation, and testing sets. For custom model training, specific splits were used (e.g., 30/5/18 for HN). Vendor-pretrained models were trained on a combination of publicly available datasets and data from vendor-collaborating clinics.

Key Insights and Ideas

  • Institutional Practices Matter: The study strongly suggests that the specific ways clinicians contour organs and targets within an institution significantly impact the performance of DL segmentation models. Vendor models, trained on broader datasets, may not capture these nuances.
  • Customization is Key for Accuracy: Custom-trained models, leveraging local data, demonstrated superior performance in many cases, particularly for structures where institutional preferences diverge from general guidelines.
  • Small Data, Big Impact: A relatively small amount of institutional data (around 30 samples) was found to be sufficient to train custom models that perform comparably to, or even better than, vendor-pretrained models. This democratizes the use of advanced DL tools for institutions with limited data.
  • Beyond Geometry: Dosimetric Impact: The differences in segmentation between vendor and custom models translated into measurable differences in radiation dose delivered to critical structures, highlighting the clinical relevance of accurate auto-segmentation.
  • Handling Novelties: Custom models showed a better ability to handle specific clinical scenarios, such as the presence of rectal spacers in prostate cancer patients, which vendor models, lacking exposure to such data, struggled with.

Key Results

  • Overall Performance: Custom-trained models achieved higher average DSCs and lower average MSDs across all three cancer sites compared to vendor-pretrained models.
  • HN: DSC 0.86 vs. 0.81, MSD 0.76 mm vs. 0.81 mm
    • Breast: DSC 0.80 vs. 0.67, MSD 2.42 mm vs. 4.85 mm
    • Prostate: DSC 0.92 vs. 0.87, MSD 1.21 mm vs. 2.17 mm
  • Structure-Specific Improvements: Custom models showed significant improvements for 14 out of 24 OARs.
  • Dosimetric Gains: Custom models generally led to smaller errors in mean and maximum dose delivery (ADmean and ADmax) for several critical structures, indicating better alignment with clinical goals.
  • Efficiency: The average prediction time for custom-trained models was less than 30 seconds per patient, comparable to vendor models.

Limitations and Future Work

  • Absence of Subjective Evaluation: The study lacked subjective evaluation by physicians, which could provide additional insights into clinical acceptability.
  • Optimal Data Size: The study did not definitively determine the optimal number of patients required for custom training, though it suggested that a modest amount is sufficient.
  • Generalizability: Custom models trained on highly consistent institutional data might underperform in cases with uncommon anatomical variations. Comprehensive quality assurance (QA) is therefore crucial.
  • Future Directions: The authors suggest that fine-tuning vendor-pretrained models using transfer learning could be a generalized approach for custom model development, especially given the effort involved in curating large datasets. Further research into the optimal number of training samples and the impact of data quality on model performance is also warranted.

Summary updated

Abstract

These findings underscore the substantial impact of institutional preferences and clinical practices on the implementation of vendor-pretrained models. We also found that a relatively small amount of institutional data was sufficient to train customized segmentation models with sufficient accuracy.
BibTeX
@inproceedings{chen2024comparison,
  author = {Xinru Chen and Yao Zhao and Hana Baroudi and Mohammad D. El Basha and Aji Daniel and Skylar Gay and Cenji Yu and He Wang and Jack Phan and Seungtaek Choi and Chelain R. Goodman and Xiaodong Zhang and Joshua S. Niedzielski and Sanjay Shete and Laurence E. Court and Zhongxing Liao and Fredrik Löfman and Peter Balter and Jinzhong Yang},
  title = {{Comparison of Vendor-Pretrained and Custom-Trained Deep Learning Segmentation Models for Head-and-Neck, Breast, and Prostate Cancers}},
  booktitle = {Diagnostics},
  year = {2024},
  doi = {10.3390/diagnostics14242851},
}
Figure