Abstract
We systematically evaluate a Deep Learning (DL) method in a 3D medical image segmentation task. Our segmentation method is integrated into the radiosurgery treatment process and directly impacts the clinical workflow. With our method, we address the relative drawbacks of manual segmentation: high inter-rater contouring variability and high time consumption of the contouring process. The main extension over the existing evaluations is the careful and detailed analysis that could be further generalized on other medical image segmentation tasks. Firstly, we analyze the changes in the inter-rater detection agreement. We show that the segmentation model reduces the ratio of detection disagreements from 0.162 to 0.085 (p < 0.05). Secondly, we show that the model improves the inter-rater contouring agreement from 0.845 to 0.871 surface Dice Score (p < 0.05). Thirdly, we show that the model accelerates the delineation process in between 1.6 and 2.0 times (p < 0.05). Finally, we design the setup of the clinical experiment to either exclude or estimate the evaluation biases, thus preserve the significance of the results. Besides the clinical evaluation, we also summarize the intuitions and practical ideas for building an efficient DL-based model for 3D medical image segmentation.
Introduction
In stereotactic radiosurgery (SRS), precision is paramount. Clinicians must deliver high-dose radiation to brain metastases with sub-millimeter accuracy while sparing surrounding healthy tissue. Currently, the most significant bottleneck in this workflow is manual contouring—the process of tracing tumor boundaries on MRI scans. This manual approach suffers from two primary limitations: high subjectivity, where different experts disagree on tumor margins, and low efficiency, as tracing multiple small lesions slice-by-slice is exceptionally time-consuming.
While many AI models demonstrate high performance on public benchmarks, standard metrics like the Dice score often fail to reflect actual clinical utility. This study bridges the gap between computer science benchmarks and clinical reality by evaluating a 3D Deep Learning segmentation tool within a live radiosurgery workflow.
Methods and Architecture
The researchers developed a customized 3D U-Net architecture tailored for the specific challenges of brain metastases. To handle the high resolution and varying sizes of lesions, the model incorporates several practical modifications:
- Residual Blocks: Integrated to improve gradient flow and overall segmentation performance.
- Non-Uniform Tumor Sampling: Instead of random sampling, the training process ensures small tumors are represented equally, preventing the model from ignoring tiny lesions.
- Inverse Weighting: The Binary Cross-Entropy loss function was modified to assign higher weights to smaller lesions, further improving detection sensitivity.
To evaluate the model, a "crossover" clinical experiment was designed with four human experts. They processed 20 patients divided into groups to account for "memory bias"—the tendency of a doctor to remember a scan they recently saw. This allowed the team to statistically isolate the benefit of AI-Assisted Contouring (AC) from the bias of repetition compared to standard Manual Contouring (MC).
Clinical Evaluation and Results
The study utilized a development dataset of 1,420 unique patients from the Moscow Gamma Knife Center and a separate clinical evaluation set of 20 unseen patients. The results demonstrated statistically significant improvements across three major clinical pillars:
- Enhanced Detection: The AI acts as a "second pair of eyes," finding lesions humans might overlook. The ratio of detection disagreements dropped significantly from 0.162 to 0.085.
- Increased Consistency: Inter-rater agreement, measured by the Surface Dice Score, improved from 0.845 to 0.871. As shown in the visual comparisons, AI assistance helps align the boundaries drawn by different experts.
- Workflow Acceleration: The AI-assisted workflow was 1.6 to 2.0 times faster than manual contouring, saving an average of 3.5 to 5 minutes per patient.
Discussion and Conclusion
A key insight from the study is the role of AI as a "leveler." While the most experienced raters were least influenced by the model, less experienced clinicians saw the greatest gains in consistency, bringing their performance closer to expert levels. By addressing the "memory effect" through rigorous experimental design, the authors confirmed that these gains are a direct result of the DL tool rather than simple repetition.
While the study is limited by a small cohort of 20 cases and four raters, the statistical significance of the results highlights the potential for DL to transform radiosurgery. Future work will focus on multi-institutional training to reduce "style bias" and ensure the model reflects a broader clinical consensus.
Summary updated