Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs

Rebecca M. Jones, Anuj Sharma, Robert Hotchkiss, John W. Sperling, Jackson Hamburger, Christian Ledig, Robert O'Toole, Michael Gardner, Srivas Venkatesh, Matthew M. Roberts, Romain Sauvestre, Max Shatkhin, Anant Gupta, Sumit Chopra, Manickam Kumaravel, Aaron Daluiski, Will Plogger, Jason Nascone, Hollis Potter, Robert V. LindseyNpj Digit. Medicine 2020 · DOI

TL;DR

This paper introduces a deep-learning system designed to detect fractures across the entire adult musculoskeletal system from radiographs. The system demonstrates high accuracy, emulating the expertise of orthopedic surgeons and radiologists, and has the potential to significantly reduce diagnostic errors in emergency departments.

What Problem Does This Paper Address?

Missed fractures are a prevalent diagnostic error in emergency departments, leading to delayed treatment and long-term disability. This issue stems from the inherent difficulty in identifying fractures, which can appear in any bone and present with subtle variations depending on the anatomical region and imaging projection. Clinicians, especially those without specialized musculoskeletal imaging training, often struggle with this task. The paper aims to bridge this gap by developing a deep-learning system that can provide timely and accurate fracture detection expertise, thereby improving patient care and reducing diagnostic errors.

Methods

The researchers developed a deep-learning system that utilizes an ensemble of 10 convolutional neural networks (CNNs). Each network in the ensemble is a variant of the Dilated Residual Network architecture. The system processes radiographs through a three-stage pipeline:

  1. Pre-processing: Radiographs are standardized by cropping, resizing (preserving aspect ratio), and optionally normalizing contrast.
  2. Analysis: The pre-processed radiographs are fed into the ensemble of CNNs. Each network predicts a "radiograph-level probability" of fracture and, for some, a "pixel-wise probability map" indicating fracture locations. The outputs from the ensemble are averaged to produce a final prediction.
  3. Post-processing: The averaged outputs are used to generate a binary determination (fracture present/absent) and bounding boxes around suspected fracture sites. Anatomical-region-specific thresholds are used for binarization to ensure adequate sensitivity across different regions.

The system was trained on a large dataset of manually annotated radiographs and then evaluated on a separate test dataset.

Datasets

  • Development Dataset: This dataset comprised 715,343 de-identified radiographs from 314,866 patients across 15 hospitals and outpatient care centers. It was manually annotated by 18 orthopedic surgeons and 11 radiologists. This dataset was split into training (80%), tuning (10%), and validation (10%) sets.
  • Test Dataset: A holdout set of 16,019 de-identified radiographs from 12,746 adults across 15 hospitals and outpatient care centers was used for evaluation. This dataset was distinct from the development set. Each radiograph in the test set was independently annotated by three orthopedic surgeons or radiologists.

Key Insights and Ideas

  • Emulating Expert Performance: The deep-learning system successfully emulates the diagnostic capabilities of experienced orthopedic surgeons and radiologists, demonstrating a high level of agreement with their judgments.
  • Broad Anatomical Coverage: Unlike previous systems limited to specific bones or regions, this model covers all 16 major anatomical regions of the musculoskeletal system, addressing a significant unmet need.
  • Robustness to Challenging Cases: The system shows robust performance even on fracture types that are typically more difficult for humans to detect, such as those without lucent lines.
  • Clinical Utility of Bounding Boxes: The decision to output bounding boxes, rather than just probability maps, was driven by physician feedback, highlighting the importance of interpretable and actionable outputs in clinical AI tools.

Key Results

The deep-learning system achieved a high overall Area Under the Curve (AUC) of 0.974 (95% CI: 0.971–0.977) on the test dataset.

  • Sensitivity: 95.2% (95% CI: 94.2–96.0%)
  • Specificity: 81.3% (95% CI: 80.7–81.9%)
  • Positive Predictive Value (PPV): 47.4% (95% CI: 46.0–48.9%)
  • Negative Predictive Value (NPV): 99.0% (95% CI: 98.8–99.1%)

Performance varied by anatomical region, with the foot showing the lowest AUC (0.888) and the knee the highest (0.996). Fracture type also influenced performance, with fractures without lucent lines having the lowest AUC (0.948) and those without callus formations the highest (0.982).

Limitations and Future Work

  • PPV: The study acknowledges that the PPV is relatively low, which is partly due to the low prevalence of fractures in the test dataset and the system's high sensitivity. Future research could explore different sensitivity-PPV balances.
  • Generalizability: While tested across two healthcare systems, further evaluation with additional healthcare systems is necessary to confirm generalizability.
  • Anatomical Region Representation: The development dataset over-represented less frequently acquired regions to ensure sufficient data for training. This might mean the overall AUC could differ in practice depending on the distribution of anatomical regions encountered.
  • Future Directions: The authors suggest further clinical research to evaluate the system's potential to reduce diagnostic errors and improve patient outcomes. The system has already received FDA clearance for a subset of anatomical regions.

Summary updated

BibTeX
@article{jones2020assessment,
  author = {Rebecca M. Jones and Anuj Sharma and Robert Hotchkiss and John W. Sperling and Jackson Hamburger and Christian Ledig and Robert O'Toole and Michael Gardner and Srivas Venkatesh and Matthew M. Roberts and Romain Sauvestre and Max Shatkhin and Anant Gupta and Sumit Chopra and Manickam Kumaravel and Aaron Daluiski and Will Plogger and Jason Nascone and Hollis Potter and Robert V. Lindsey},
  title = {{Assessment of a deep-learning system for fracture detection in musculoskeletal radiographs}},
  journal = {Npj Digit. Medicine},
  year = {2020},
  doi = {10.1038/s41746-020-00352-w},
}
Figure