Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19

Zaid Nabulsi, Andrew Sellergren, Shahar Jamshy, Charles T. Lau, Edward Santos, Atilla P. Kiraly, Wenxing Ye, Jie Yang, Rory Pilgrim, Sahar Kazemzadeh, Yu Jin, Sreenivasa Raju Kalidindi, Mozziyar Etemadi, Florencia Garcia-Vicente, David Melnick, Greg S. Corrado, Lily Peng, Krishnan Eswaran, Daniel Tse, Neeral Beladia, Yun Liu, Po-Hsuan Cameron Chen, Shravya Shetty

TL;DR

This paper introduces a deep learning system that classifies chest X-rays (CXRs) as either normal or abnormal, demonstrating its ability to generalize to new patient populations and even previously unseen diseases like tuberculosis and COVID-19. This approach shows promise for improving radiologist workflow by prioritizing potentially abnormal cases, leading to significant reductions in turnaround time.

What Problem Does This Paper Address?

Interpreting chest X-rays (CXRs) is a cornerstone of diagnosing thoracic conditions, but the sheer volume of these scans creates a significant workload for radiologists. While AI has shown success in detecting specific abnormalities, building separate models for every possible condition is impractical due to the vast range of potential findings and the need for large, labeled datasets for each. This paper tackles the challenge of creating a more generalizable AI system that can broadly distinguish between normal and abnormal CXRs, acting as an effective triage tool to help radiologists manage their workload more efficiently, especially in high-demand situations or when encountering novel diseases.

Methods

The core of this work is a deep learning system (DLS) built using a convolutional neural network (CNN), specifically leveraging the EfficientNet-B7 architecture pre-trained on natural images. To adapt it for CXR analysis, the single-channel grayscale CXR images were tiled into three channels. The CNN was trained using cross-entropy loss and a momentum optimizer. Crucially, the DLS was trained to output a continuous score indicating the likelihood of an abnormality, rather than specific diagnoses. For clinical decision-making, operating points were established to convert these scores into binary "normal" or "abnormal" classifications, with specific points chosen for high sensitivity (to rule out normal cases) and high specificity (to confirm abnormalities). The model was trained on a large dataset and then rigorously evaluated on multiple independent datasets, including those with diseases it had not been exposed to during training.

Datasets

The study utilized a multi-stage approach with several datasets:

  • Training and Tuning: A large, de-identified dataset (DS-1) comprising 248,445 CXRs from 213,889 patients across five hospital clusters in India was used for initial model development. This dataset excluded known tuberculosis (TB) cases and COVID-19 cases.
  • Evaluation: Six independent international datasets were used to assess the DLS's generalizability:
  • General Abnormalities: DS-1 (test split) and the publicly available ChestX-ray14 (CXR-14) dataset.
  • Unseen Diseases (TB): Two datasets (TB-1 and TB-2) containing CXRs from patients with microbiologically confirmed TB.
  • Unseen Diseases (COVID-19): Two datasets (COV-1 and COV-2) containing CXRs from patients with RT-PCR-confirmed COVID-19.

The paper also mentions the release of abnormal vs. normal labels from three radiologists for the publicly available CXR-14 test set to aid future research.

Key Insights and Ideas

  • Generalization is Key: The most significant insight is that a DLS trained to broadly classify CXRs as normal or abnormal can generalize well to new patient populations and, importantly, to diseases it was not explicitly trained on. This moves beyond the limitations of highly specific diagnostic AI.
  • Triage Power: The DLS shows strong potential as a triage tool. By prioritizing potentially abnormal cases, it can significantly reduce the time radiologists spend reviewing scans, freeing them up for more complex cases or increasing overall throughput.
  • Operating Point Matters: The performance of the DLS is highly dependent on the chosen operating point (sensitivity vs. specificity). Selecting appropriate operating points for different clinical scenarios (e.g., high sensitivity for screening, high specificity for confirmation) is crucial for its effective deployment.
  • Unseen Diseases are a Challenge, but Solvable: While performance on unseen diseases like COVID-19 was lower than on general abnormalities, the DLS still demonstrated meaningful performance, highlighting the potential for such general classifiers to adapt. The challenges observed with COVID-19 were partly attributed to atypical presentations and the presence of other abnormalities in "negative" cases.

Key Results

  • Generalizability: The DLS achieved good performance on general abnormality detection, with AUCs of 0.87 for DS-1 and 0.94 for CXR-14.
  • Unseen Diseases: The DLS showed strong performance on TB datasets (AUCs of 0.95 and 0.97) and moderate performance on COVID-19 datasets (AUCs of 0.68 and 0.65).
  • Workflow Improvement: In a simulated workflow, DLS-based prioritization of abnormal cases reduced the mean turnaround time by 7-28% across different datasets.
  • Radiologist Comparison: At a high-sensitivity operating point, the DLS's negative predictive values (NPVs) for ruling out normal CXRs were comparable to those of board-certified radiologists.

Limitations and Future Work

  • Limited Disease Representation: The study acknowledges that a wide range of abnormalities and diseases were not represented in the available CXRs, limiting the exhaustive evaluation of the DLS. Future work could involve expanding the dataset to include more conditions.
  • "Negative" Cases with Other Abnormalities: For TB and COVID-19, "negative" cases (i.e., not having the specific disease) could still contain other abnormalities, complicating the interpretation of metrics like PPV.
  • Retrospective Data and Simulation: The study relied on retrospective data and simulated workflows. The true impact on clinical practice needs to be validated in prospective settings.
  • Clinical Context: Radiologists in real-world settings often have access to clinical and patient data, which was not provided to the DLS or the radiologists in this study. Incorporating such information could further enhance performance.
  • Site-Specific Operating Points: While operating points generalized well between similar datasets, the authors suggest that site-specific operating point selection could further improve performance, indicating a need for adaptive AI systems.
  • Data Shifts: The study observed distributional shifts between datasets, emphasizing the importance of scenario-specific operating points and potentially site-specific tuning.

Summary updated

Abstract

Chest radiography (CXR) is the most widely-used thoracic clinical imaging modality and is crucial for guiding the management of cardiothoracic conditions. The detection of specific CXR findings has been the main focus of several artificial intelligence (AI) systems. However, the wide range of possible CXR abnormalities makes it impractical to detect every possible condition by building multiple separate systems, each of which detects one or more pre-specified conditions. In this work, we developed and evaluated an AI system to classify CXRs as normal or abnormal. For training and tuning the system, we used a de-identified dataset of 248,445 patients from a multi-city hospital network in India. To assess generalizability, we evaluated our system using 6 international datasets from India, China, and the United States. Of these datasets, 4 focused on diseases that the AI was not trained to detect: 2 datasets with tuberculosis and 2 datasets with coronavirus disease 2019. Our results suggest that the AI system trained using a large dataset containing a diverse array of CXR abnormalities generalizes to new patient populations and unseen diseases. In a simulated workflow where the AI system prioritized abnormal cases, the turnaround time for abnormal cases reduced by 7-28%. These results represent an important step towards evaluating whether AI can be safely used to flag cases in a general setting where previously unseen abnormalities exist. Lastly, to facilitate the continued development of AI models for CXR, we release our collected labels for the publicly available dataset.
BibTeX
@inproceedings{nabulsi2021deep,
  author = {Zaid Nabulsi and Andrew Sellergren and Shahar Jamshy and Charles T. Lau and Edward Santos and Atilla P. Kiraly and Wenxing Ye and Jie Yang and Rory Pilgrim and Sahar Kazemzadeh and Yu Jin and Sreenivasa Raju Kalidindi and Mozziyar Etemadi and Florencia Garcia-Vicente and David Melnick and Greg S. Corrado and Lily Peng and Krishnan Eswaran and Daniel Tse and Neeral Beladia and Yun Liu and Po-Hsuan Cameron Chen and Shravya Shetty},
  title = {{Deep learning for distinguishing normal versus abnormal chest radiographs and generalization to two unseen diseases tuberculosis and COVID-19}},
  booktitle = {Scientific},
  year = {2021},
  doi = {10.1038/s41598-021-93967-2},
}
Figure