A Deep Learning Model to Triage Screening Mammograms: A Simulation Study

Adam Yala, Tal Schuster, Randy C. Miles, Regina Barzilay, Constance D. LehmanRadiology 2019 · DOI

TL;DR

This paper introduces a deep learning (DL) model designed to triage screening mammograms, identifying a subset as likely cancer-free. This approach significantly reduces radiologist workload and improves specificity without compromising sensitivity, offering a promising avenue for more efficient breast cancer screening.

What Problem Does This Paper Address?

Mammography is a critical tool for early breast cancer detection, but its effectiveness is hampered by variations in radiologist performance and workflow inefficiencies. Specifically, radiologists face a high volume of screening mammograms, leading to potential fatigue and errors. While existing computer-aided detection (CAD) systems aim to improve sensitivity, they often don't address radiologist specificity or overall workflow efficiency. Double reading, another method to improve accuracy, further burdens the workflow. This paper tackles the challenge of improving both the efficiency and accuracy of mammogram interpretation by developing a DL model that can confidently identify "cancer-free" cases, allowing radiologists to focus their expertise on more complex or suspicious scans.

Methods

The researchers developed a deep learning model based on a convolutional neural network (ResNet18) trained to predict the likelihood of a breast developing cancer within one year. The model was trained on a large dataset of screening mammograms and their corresponding cancer outcomes. For the triage application, a high-sensitivity threshold was established on the validation set. This threshold was set to the minimum probability score that corresponded to a radiologist's true-positive assessment. In a simulated workflow, mammograms falling below this threshold were triaged as "cancer-free" and skipped by radiologists. Mammograms above the threshold were then read by radiologists as usual. The study then compared the performance metrics (sensitivity, specificity, and percentage of mammograms read) of this simulated DL-triage workflow against the standard workflow where radiologists read all mammograms.

Datasets

The study utilized a retrospective dataset of 223,109 consecutive screening mammograms from 66,661 women, collected between January 2009 and December 2016. This dataset was split into training (212,272 mammograms from 56,831 patients), validation (25,999 mammograms from 7,021 patients), and testing (26,540 mammograms from 7,176 patients) sets. Cancer outcomes were obtained through linkage to regional tumor registries and pathology reports. The dataset was carefully curated to include women with either a diagnosis of breast cancer within one year or at least one year of imaging follow-up.

Key Insights and Ideas

  • Shifting Focus from Detection to Triage: Instead of solely aiming to "detect" suspicious findings like traditional CAD, this DL model focuses on "triaging" – confidently identifying likely negative cases to reduce workload.
  • Leveraging "Cancer-Free" Confidence: The model's strength lies in its ability to achieve high confidence in identifying mammograms that are very unlikely to harbor cancer, enabling a significant portion of the workload to be bypassed by human readers.
  • Complementary to CAD: This triage approach is presented as complementary to CAD systems, potentially working together to optimize the screening process.
  • Broad Applicability: The model demonstrated consistent performance across various demographic groups (age, race) and breast densities, suggesting its potential for widespread use.

Key Results

In the simulated DL-triage workflow:

  • The model triaged approximately 19.3% of mammograms as cancer-free, reducing the number of mammograms read by radiologists.
  • Specificity improved significantly from 93.5% to 94.2% (P = .002).
  • Sensitivity remained non-inferior, with a slight decrease from 90.6% to 90.1% (P < .001), well within the 5% margin.
  • The workload reduction was substantial, with radiologists reading only 80.7% of mammograms.
  • The model showed similar predictive accuracy across different age groups and breast densities, with AUCs generally ranging from 0.71 to 0.86.

Limitations and Future Work

The authors acknowledge several limitations:

  • Retrospective Simulation: The study relied on a retrospective simulation, assuming radiologists would interpret remaining images the same way. A prospective trial is needed to confirm these findings in real-world clinical practice.
  • Single Institution and Vendor: The model was developed and tested at a single academic institution using mammograms from a single vendor (Hologic). External validation with diverse populations and mammography equipment is crucial.
  • Potential for Radiologist Bias: The knowledge of risk assessment and reduced workload might influence radiologist performance in ways not captured by this simulation.

Future work should focus on prospective validation, exploring different triage thresholds, and investigating the model's performance with mammograms from diverse vendors. The authors also suggest that this model could be adapted for scenarios with limited access to specialized radiologists or for identifying cases suitable for single-human assessment in regions where double reading is common.

Summary updated

Abstract

Background Recent deep learning (DL) approaches have shown promise in improving sensitivity but have not addressed limitations in radiologist specificity or efficiency. Purpose To develop a DL model to triage a portion of mammograms as cancer free, improving performance and workflow efficiency. Materials and Methods In this retrospective study, 223 109 consecutive screening mammograms performed in 66 661 women from January 2009 to December 2016 were collected with cancer outcomes obtained through linkage to a regional tumor registry. This cohort was split by patient into 212 272, 25 999, and 26 540 mammograms from 56 831, 7021, and 7176 patients for training, validation, and testing, respectively. A DL model was developed to triage mammograms as cancer free and evaluated on the test set. A DL-triage workflow was simulated in which radiologists skipped mammograms triaged as cancer free (interpreting them as negative for cancer) and read mammograms not triaged as cancer free by using the original interpreting radiologists' assessments. Sensitivities, specificities, and percentage of mammograms read were calculated, with and without the DL-triage-simulated workflow. Statistics were computed across 5000 bootstrap samples to assess confidence intervals (CIs). Specificities were compared by using a two-tailed <i>t</i> test (<i>P</i> < .05) and sensitivities were compared by using a one-sided <i>t</i> test with a noninferiority margin of 5% (<i>P</i> < .05). Results The test set included 7176 women (mean age, 57.8 years ± 10.9 [standard deviation]). When reading all mammograms, radiologists obtained a sensitivity and specificity of 90.6% (173 of 191; 95% CI: 86.6%, 94.7%) and 93.5% (24 625 of 26 349; 95% CI: 93.3%, 93.9%). In the DL-simulated workflow, the radiologists obtained a sensitivity and specificity of 90.1% (172 of 191; 95% CI: 86.0%, 94.3%) and 94.2% (24 814 of 26 349; 95% CI: 94.0%, 94.6%) while reading 80.7% (21 420 of 26 540) of the mammograms. The simulated workflow improved specificity (<i>P</i> = .002) and obtained a noninferior sensitivity with a margin of 5% (<i>P</i> < .001). Conclusion This deep learning model has the potential to reduce radiologist workload and significantly improve specificity without harming sensitivity. © RSNA, 2019 <i>Online supplemental material is available for this article.</i> See also the editorial by Kontos and Conant in this issue.
BibTeX
@inproceedings{yala2019deep,
  author = {Adam Yala and Tal Schuster and Randy C. Miles and Regina Barzilay and Constance D. Lehman},
  title = {{A Deep Learning Model to Triage Screening Mammograms: A Simulation Study}},
  booktitle = {Radiology},
  year = {2019},
  doi = {10.1148/radiol.2019182908},
}
Figure