Automated segmentation of brain metastases with deep learning: A multi-center, randomized crossover, multi-reader evaluation study

Xiao Luo, Yadi Yang, Shaohan Yin, Hui Li, Ying Shao, Dechun Zheng, Xinchun Li, Jianpeng Li, Weixiong Fan, Jing Li, Xiaohua Ban, Shan-Shan Lian, Yun Zhang, Qiuxia Yang, Weijing Zhang, Cheng Zhang, Lidi Ma, Yingwei Luo, Fan Zhou, Shiyuan Wang, Cuiping Lin, Jiao Li, Ma Luo, Jian-xun He, Guixiao Xu, Yaozong Gao, Dinggang Shen, Ying Sun, Yonggao Mou, Rong Zhang, Chuanmiao XieNeuro-Oncology 2024 · DOI

TL;DR

This paper introduces a deep learning system (BMSS) for automatically segmenting brain metastases (BMs) from MRI scans. In a multi-center, randomized reader study, the BMSS not only achieved high accuracy but also significantly improved the efficiency of radiologists, reducing contouring time by 42% while maintaining or improving accuracy.

What Problem Does This Paper Address?

Accurate segmentation of brain metastases (BMs) from MRI scans is crucial for effective treatment planning, particularly for stereotactic radiosurgery (SRS). However, manual contouring is a labor-intensive, time-consuming, and highly variable process among radiologists. While deep learning (DL) has shown promise for automated segmentation, its real-world clinical utility, generalizability across different centers, and impact on radiologist workflow have remained largely unvalidated. This study aims to bridge this gap by developing and rigorously evaluating a DL-based BM segmentation system (BMSS) in a multi-center, prospective, and randomized reader study.

Methods

The core of the study is the development and evaluation of a deep learning system called the Brain Metastasis Segmentation System (BMSS). The BMSS is built upon a VB-Net architecture, a 3D segmentation model derived from V-Net, which was optimized for parameter reduction and efficient deployment. The model was trained on a large dataset of over 10,000 BM lesions.

To evaluate its clinical impact, a randomized crossover, multi-reader, multi-case (MRMC) study was conducted. This involved 10 radiologists (5 residents and 5 attending radiologists) who contoured BMs both with and without the assistance of the BMSS. The study design ensured that readers were randomized to either assisted-first or unassisted-first sessions, separated by a significant washout period to minimize bias. Key metrics compared included Dice Similarity Coefficient (DSC) for accuracy, Average Surface Distance (ASD) for boundary discrepancies, and contouring time per lesion.

Datasets

  • Development Dataset: Collected from Sun Yat-sen University Cancer Center (SYSUCC), comprising 488 patients with 10,338 brain metastases. This dataset was split into training (332 patients, 8,360 BMs) and internal validation (156 patients, 1,978 BMs) sets.
  • Prospective Multi-Center Test Dataset: Collected from 5 medical centers across China, consisting of 50 patients with 203 brain metastases. This dataset was used to evaluate the model's generalizability and its impact on reader performance.

Key Insights and Ideas

  • Bridging the "Implementation Gap": The study directly addresses the critical gap between promising preclinical DL models and their actual clinical utility. By using a multi-center, prospective, and randomized reader study, it provides a robust evaluation of the BMSS in a realistic clinical setting.
  • Efficiency Gains are Significant: The BMSS doesn't just offer comparable accuracy; it dramatically improves radiologist efficiency. The 42% time saving per lesion is a substantial benefit for clinical workflows, especially in busy oncology departments.
  • Resident Radiologists Benefit Most: While both junior and senior radiologists saw improvements, resident radiologists experienced a greater leap in contouring accuracy when using the BMSS, suggesting it can be a powerful training and assistance tool for less experienced clinicians.
  • Robustness Across Centers: The BMSS demonstrated comparable performance on both internal and external multi-center datasets, indicating good generalizability, which is a crucial factor for widespread adoption.

Key Results

  • Standalone BMSS Performance: The BMSS achieved a high median Dice Similarity Coefficient (DSC) of 0.91 (95% CI, 0.90–0.92) and a median Average Surface Distance (ASD) of 0.40 mm (95% CI, 0.36–0.44) on the multi-center test dataset.
  • Impact of AI Assistance:
    • Accuracy Improvement: With BMSS assistance, readers increased their median DSC from 0.87 (unassisted) to 0.92 (assisted) (P < .001).
  • Time Saving: BMSS assistance resulted in a median time saving of 42% per lesion (40-45%).
  • Resident vs. Attending: Resident readers showed a greater improvement in accuracy (median DSC increase of 0.05 vs. 0.03 for attending radiologists, P < .001) but similar time reductions.
  • RVD Reduction: The relative volume difference (RVD) was significantly reduced by 53% with BMSS assistance, indicating more accurate volume estimations.

Limitations and Future Work

  • Limited to BMs: The current network is specifically trained for brain metastases. Future work should explore extending its capabilities to segment other types of brain tumors (e.g., glioma, meningioma).
  • Single Referral Center Recruitment: While multi-center data was used for evaluation, the radiologists participating in the reading experiment were all from a single cancer referral center. Further validation with radiologists from diverse hospital settings is needed to assess broader feasibility.
  • MRI Sequence Dependence: The model was primarily trained on gradient echo sequences, and its performance on other sequences (like fast-spin echo) needs further investigation.
  • Test Set Imbalance: The test set had a smaller sample size and larger tumor sizes compared to the training set, which could influence interpretation.
  • Irregular and Meningeal Lesions: The BMSS showed some limitations in segmenting highly irregular lesions or those adhering to the meninges, areas where human variability is also high.
  • Future Directions: The authors suggest exploring the application of Al in optimizing radiotherapy dose and fractionation regimens, and investigating how BMSS-assisted segmentation can directly benefit patient outcomes like local control and reduced radiotoxicity.

Summary updated

Abstract

The BMSS can be optimally applied to improve the efficiency of brain metastasis delineation in clinical practice.
BibTeX
@inproceedings{luo2024automated,
  author = {Xiao Luo and Yadi Yang and Shaohan Yin and Hui Li and Ying Shao and Dechun Zheng and Xinchun Li and Jianpeng Li and Weixiong Fan and Jing Li and Xiaohua Ban and Shan-Shan Lian and Yun Zhang and Qiuxia Yang and Weijing Zhang and Cheng Zhang and Lidi Ma and Yingwei Luo and Fan Zhou and Shiyuan Wang and Cuiping Lin and Jiao Li and Ma Luo and Jian-xun He and Guixiao Xu and Yaozong Gao and Dinggang Shen and Ying Sun and Yonggao Mou and Rong Zhang and Chuanmiao Xie},
  title = {{Automated segmentation of brain metastases with deep learning: A multi-center, randomized crossover, multi-reader evaluation study}},
  booktitle = {Neuro-Oncology},
  year = {2024},
  doi = {10.1093/neuonc/noae113},
}
Figure