Deep neural network to locate and segment brain tumors outperformed the expert technicians who created the training data

J. Ross Mitchell, Konstantinos Kamnitsas, Kyle Singleton, Scott A. Whitmire, Kamala Clark-Swanson, Sara Ranjbar, Cassandra Rickertsen, Sandra K. Johnston, Kathleen M. Egan, Dana E. Rollison, John A. Arrington, Karl N. Krecke, Theodore J. Passe, Jared T. Verdoorn, Alex A. Nagelschneider, Carrie M. Carr, John D. Port, Alice Patton, Norbert G. Campeau, Greta B. Liebo, Laurence J. Eckel, Christopher P. Wood, Christopher H. Hunt, Prasanna Vibhute, Kent D. Nelson, Joseph M. Hoxworth, Ameet C. Patel, Brian W. Chong, Jeffrey S. Ross, Jerrold L. Boxerman, Michael A. Vogelbaum, Leland Hu, Ben Glocker, Kristin R. SwansonJournal of Medical Imaging 2020 · DOI

TL;DR

This study presents the first blinded, randomized comparison of deep learning (DL) based brain tumor segmentation against expert human segmentations. The DL system, trained on technician-generated data, not only matched but significantly outperformed its human creators, achieving higher scores from neuroradiologists and a superior Dice coefficient. This suggests DL models can surpass the quality of their training data, offering a promising avenue for more accurate and efficient medical image analysis.

What Problem Does This Paper Address?

Accurately segmenting brain tumors in MRI scans is crucial for diagnosis, treatment planning, and monitoring. While deep learning (DL) models have shown great promise in this area, their "black box" nature makes validation and trust a significant challenge. Traditionally, DL models are trained on data meticulously labeled by human experts (radiologists or technicians). However, the quality and consistency of these human labels can vary, and it's unclear if DL models can truly learn to perform better than the experts who created their training data. This paper addresses the critical need for a rigorous, objective comparison between DL-generated segmentations and human-generated segmentations, specifically testing whether DL can outperform its "teachers."

Methods

The researchers developed a novel platform for a blinded, controlled assessment of brain tumor segmentations. This platform allowed 20 neuroradiologists to compare segmentations from both a deep learning model (DeepMedic) and trained technicians side-by-side, scoring them on a scale of 0-10. The study also quantitatively evaluated agreement using the Dice coefficient, a standard metric for segmentation overlap.

The DL model, DeepMedic, was trained on a dataset of 641 pretreatment MRI exams. The training process involved hyperparameter tuning and then training an ensemble of five networks. The key aspects of the DL approach included:

  • Input Data: T1-weighted postcontrast and T2-weighted FLAIR MRI sequences.
  • Preprocessing: A fully automated pipeline involving artifact reduction, skull-stripping, intensity normalization, and image resampling.
  • Network Architecture: A 3D convolutional neural network (DeepMedic) known for state-of-the-art performance in similar challenges.
  • Training Strategy: A combination of stochastic gradient descent with warm restarts (SGDR) and learning rate decay schedules to optimize performance.

The technician segmentations were derived from a large multi-institutional database of brain tumor MRIs, with segmentations performed by a team of 15 technicians. For scans with multiple technician segmentations, a consensus was formed.

Datasets

  • Training Data: 641 pretreatment MRI exams from a multi-institutional database.
  • Testing Data: 100 pretreatment MRI exams, distinct from the training set.
  • Overall Dataset: A total of 741 pretreatment MRI exams from 729 unique patients were compiled. This dataset included exams from eight North American academic cancer centers, three public domain datasets, and one consortium dataset, acquired between 1990 and 2016.

The study did not introduce a new benchmark dataset but rather utilized an existing collection and a novel evaluation methodology.

Key Insights and Ideas

  • DL Outperforms Human Teachers: The most striking finding is that the DL system, trained on data created by technicians, achieved significantly better scores from expert neuroradiologists than the technician-generated segmentations themselves. This challenges the notion that a model is inherently limited by the quality of its training data.
  • "Back-loading" Expertise: Instead of solely relying on expert-labeled data for training, this study "back-loads" human expertise by using it for rigorous, blinded evaluation of the DL output. This provides a more objective measure of performance.
  • Subjectivity in Human Assessment: The study highlights the inherent subjectivity in human interpretation of medical images, even among experts. While the DL system showed consistent improvement, the neuroradiologists' scores exhibited variability.
  • Potential for Improved Training Data Strategies: The findings suggest that strategies beyond simply acquiring more high-quality labels might be beneficial. Acquiring a larger number of lower-quality labels, potentially with repeated segmentations, could lead to models with better overall performance.

Key Results

  • Neuroradiologist Scores:
    • Technician segmentations: Mean score of 6.97 (±0.12).
  • DL segmentations: Mean score of 7.31 (±0.13).
  • The difference in mean scores (0.34) was statistically significant (p < 0.00007).
  • Dice Coefficient: The DL method achieved a mean Dice coefficient of 0.87 on the test cases, indicating a high degree of overlap with the ground truth (which, in this context, is implicitly the technician-generated segmentations used for evaluation).
  • Training Time: Training each of the five ensemble instances took an average of 28 hours and 51 minutes.
  • Segmentation Speed: Once trained, each DL instance could segment a brain tumor in approximately 7.91 seconds per exam.

Limitations and Future Work

  • Limitations:
    • The study focused on pretreatment MRI exams. The performance of the DL system on scans acquired during or after treatment, which can alter brain appearance, was not evaluated.
  • The dataset, while multi-institutional, might not fully represent the diversity of all MRI scanners and acquisition parameters encountered in clinical practice.
  • The study acknowledges that the "ground truth" for evaluation was based on technician segmentations, which themselves can have imperfections.
  • Future Work:
    • The authors suggest training the DL model with exams acquired throughout treatment and follow-up.
  • They propose exploring new strategies for creating training data, such as acquiring a larger number of lower-quality segmentations with repeated measurements, to potentially improve model performance.
  • Further research is needed to investigate the impact of lesion complexity, expert experience, and the number of segmentations on model performance.
  • The authors also note the potential for using fewer MRI sequences (two in their case) to make the method more practical for clinical workflows.

Summary updated

Abstract

<b>Purpose:</b> Deep learning (DL) algorithms have shown promising results for brain tumor segmentation in MRI. However, validation is required prior to routine clinical use. We report the first randomized and blinded comparison of DL and trained technician segmentations. <b>Approach:</b> We compiled a multi-institutional database of 741 pretreatment MRI exams. Each contained a postcontrast T1-weighted exam, a T2-weighted fluid-attenuated inversion recovery exam, and at least one technician-derived tumor segmentation. The database included 729 unique patients (470 males and 259 females). Of these exams, 641 were used for training the DL system, and 100 were reserved for testing. We developed a platform to enable qualitative, blinded, controlled assessment of lesion segmentations made by technicians and the DL method. On this platform, 20 neuroradiologists performed 400 side-by-side comparisons of segmentations on 100 test cases. They scored each segmentation between 0 (poor) and 10 (perfect). Agreement between segmentations from technicians and the DL method was also evaluated quantitatively using the Dice coefficient, which produces values between 0 (no overlap) and 1 (perfect overlap). <b>Results:</b> The neuroradiologists gave technician and DL segmentations mean scores of 6.97 and 7.31, respectively ( <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mrow><mml:mi>p</mml:mi> <mml:mo><</mml:mo> <mml:mn>0.00007</mml:mn></mml:mrow> </mml:math> ). The DL method achieved a mean Dice coefficient of 0.87 on the test cases. <b>Conclusions:</b> This was the first objective comparison of automated and human segmentation using a blinded controlled assessment study. Our DL system learned to outperform its "human teachers" and produced output that was better, on average, than its training data.
BibTeX
@article{mitchell2020deep,
  author = {J. Ross Mitchell and Konstantinos Kamnitsas and Kyle Singleton and Scott A. Whitmire and Kamala Clark-Swanson and Sara Ranjbar and Cassandra Rickertsen and Sandra K. Johnston and Kathleen M. Egan and Dana E. Rollison and John A. Arrington and Karl N. Krecke and Theodore J. Passe and Jared T. Verdoorn and Alex A. Nagelschneider and Carrie M. Carr and John D. Port and Alice Patton and Norbert G. Campeau and Greta B. Liebo and Laurence J. Eckel and Christopher P. Wood and Christopher H. Hunt and Prasanna Vibhute and Kent D. Nelson and Joseph M. Hoxworth and Ameet C. Patel and Brian W. Chong and Jeffrey S. Ross and Jerrold L. Boxerman and Michael A. Vogelbaum and Leland Hu and Ben Glocker and Kristin R. Swanson},
  title = {{Deep neural network to locate and segment brain tumors outperformed the expert technicians who created the training data}},
  journal = {Journal of Medical Imaging},
  year = {2020},
  doi = {10.1117/1.jmi.7.5.055501},
}
Figure