Systematic Clinical Evaluation of a Deep Learning Method for Medical Image Segmentation: Radiosurgery Application

Boris Shirokikh, Alexandra Dalechina, Alexey Shevtsov, Egor Krivov, Valery Kostjuchenko, Amayak Durgaryan, Mikhail Galkin, Andrey Golanov, Mikhail Belyaev

Abstract

We systematically evaluate a Deep Learning model in a 3D medical image segmentation task. With our model, we address the flaws of manual segmentation: high inter-rater contouring variability and time consumption of the contouring process. The main extension over the existing evaluations is the careful and detailed analysis that could be further generalized on other medical image segmentation tasks. Firstly, we analyze the changes in the inter-rater detection agreement. We show that the model reduces the number of detection disagreements by 48% (p < 0.05). Secondly, we show that the model improves the inter-rater contouring agreement from 0.845 to 0.871 surface Dice Score (p < 0.05). Thirdly, we show that the model accelerates the delineation process between 1.6 and 2.0 times (p < 0.05). Finally, we design the setup of the clinical experiment to either exclude or estimate the evaluation biases; thus, preserving the significance of the results. Besides the clinical evaluation, we also share intuitions and practical ideas for building an efficient DL-based model for 3D medical image segmentation.

Introduction

Stereotactic radiosurgery (SRS) is a highly precise form of radiation therapy used to treat brain metastases. The success of SRS depends heavily on the accurate delineation of tumor boundaries, a process known as "contouring." Currently, this is performed manually by clinicians on MRI scans. However, manual contouring suffers from two critical drawbacks: high inter-rater variability (subjectivity) and significant time consumption, especially when dealing with multiple small lesions. While many deep learning (DL) models have been proposed to automate this task, few have undergone rigorous clinical evaluation to measure their impact on the human-in-the-loop workflow.

Methods

Model Architecture

The researchers developed a customized 3D U-Net optimized for medical imaging constraints. Key modifications include the integration of Residual Blocks to improve gradient flow and the use of channel-wise summation instead of concatenation in skip connections. This latter change significantly reduces memory overhead, allowing the model to run on standard clinical hardware with 6GB GPUs.

The customized 3D U-Net architecture featuring residual blocks and memory-efficient skip connections.

To address the challenge of small lesions, the team implemented a non-uniform patch sampling strategy during training, ensuring the model encountered enough small metastases to avoid bias toward larger targets. Additionally, an inverse weighting loss function was employed to penalize errors on small lesions more heavily.

Clinical Experiment Design

The study utilized a development dataset of 1,420 patients and a separate evaluation set of 20 patients with 115 total lesions. To rigorously test the model, four experts with varying experience levels performed both manual and AI-assisted contouring. A crossover design was used to mitigate "memory bias," where raters might remember a scan from a previous session.

Results and Discussion

AI as a Detection Aid

A primary finding of the study is that AI serves as a critical safety net for detection. By highlighting potential targets, the model reduced the number of detection disagreements between experts by 48% ($p < 0.05$). This is particularly vital for small metastases, which are easily overlooked during manual review.

Distribution of rater agreement based on lesion diameter, highlighting the prevalence of disagreement in smaller metastases.

Consistency and Efficiency

The model significantly improved the consistency of boundaries drawn by different doctors. The inter-rater agreement, measured by the Surface Dice Score (sDSC), rose from 0.845 to 0.871. The authors advocate for sDSC over standard Dice metrics as it is more sensitive to the boundary disagreements that characterize clinical subjectivity.

Furthermore, the AI-assisted workflow was 1.6 to 2.0 times faster than manual contouring, saving approximately 5 minutes of high-concentration work per patient. On the hold-out test set, the model achieved a 0.9 recall with only 3.8 false positives per image.

Conclusion and Future Work

This systematic evaluation demonstrates that DL models can provide substantial clinical value by reducing human error and increasing efficiency in radiosurgery planning. While the results are statistically significant, the authors note that future work should involve larger, multi-center studies to test generalization across different MRI scanner manufacturers and protocols. Additionally, extending the "washout" period between sessions could further minimize memory bias in clinical trials.

Summary updated

Abstract

We systematically evaluate a Deep Learning model in a 3D medical image segmentation task. With our model, we address the flaws of manual segmentation: high inter-rater contouring variability and time consumption of the contouring process. The main extension over the existing evaluations is the careful and detailed analysis that could be further generalized on other medical image segmentation tasks. Firstly, we analyze the changes in the inter-rater detection agreement. We show that the model reduces the number of detection disagreements by <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text{48}\%$</tex-math></inline-formula> <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {(p &lt; 0.05)}$</tex-math></inline-formula> . Secondly, we show that the model improves the inter-rater contouring agreement from <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {0.845}$</tex-math></inline-formula> to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {0.871}$</tex-math></inline-formula> surface Dice Score <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {(p &lt; 0.05)}$</tex-math></inline-formula> . Thirdly, we show that the model accelerates the delineation process between <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {1.6}$</tex-math></inline-formula> and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {2.0}$</tex-math></inline-formula> times <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><tex-math notation="LaTeX">$\text {(p &lt; 0.05)}$</tex-math></inline-formula> . Finally, we design the setup of the clinical experiment to either exclude or estimate the evaluation biases; thus, preserving the significance of the results. Besides the clinical evaluation, we also share intuitions and practical ideas for building an efficient DL-based model for 3D medical image segmentation.
BibTeX
@article{shirokikh2022systematic,
  author = {Boris Shirokikh and Alexandra Dalechina and Alexey Shevtsov and Egor Krivov and Valery Kostjuchenko and Amayak Durgaryan and Mikhail Galkin and Andrey Golanov and Mikhail Belyaev},
  title = {{Systematic Clinical Evaluation of a Deep Learning Method for Medical Image Segmentation: Radiosurgery Application}},
  journal = {IEEE J. Biomed. Health Informatics},
  year = {2022},
  url = {https://openreview.net/forum?id=RMOuaG6NIj6},
}
Figure