Development and validation of an abnormality-derived deep-learning diagnostic system for major respiratory diseases

Chengdi Wang, Jiechao Ma, Shu Zhang, Jun Shao, Yanyan Wang, Hong-Yu Zhou, Lujia Song, Jie Zheng, Yizhou Yu, Weimin LiNpj Digit. Medicine 2022 · DOI

TL;DR

This paper introduces DeepMRDTR, a novel deep learning system that can identify a wide range of abnormalities and diagnose major respiratory diseases from CT and X-ray images. It achieves high accuracy, comparable to senior experts, and has been integrated into a clinical workflow, demonstrating its potential to accelerate diagnosis and improve efficiency in respiratory care.

What Problem Does This Paper Address?

Respiratory diseases are a significant global health burden, often difficult to diagnose due to the complexity of imaging findings and the uneven distribution of medical resources. While AI has shown promise in medical image analysis, existing systems often focus on single diseases or a limited set of abnormalities, lacking the comprehensive diagnostic capability needed for real-world clinical practice. This paper addresses the need for a robust, automated system that can accurately identify multiple abnormalities and diagnose a broad spectrum of major respiratory diseases from both CT and X-ray images, thereby assisting clinicians and improving diagnostic efficiency, especially in resource-constrained settings.

Methods

The core of this work is the DeepMRDTR system, a deep learning framework comprising four networks: two for CT images (CT-Nets) and two for X-ray images (CXR-Nets).

  • Abnormality Identification: Separate CT-Nets and CXR-Nets are trained to identify approximately 20 different radiological abnormalities.
  • Disease Diagnosis: A two-stream network architecture is employed for disease diagnosis. This approach first extracts features related to abnormalities and then fuses them with other learned features to predict one of eight major respiratory diseases. This "abnormality-derived" approach aims to mimic the human diagnostic process.
  • Contrastive Learning: To enhance the efficiency of transfer learning and improve model performance, especially with limited labeled data, the researchers utilized novel contrastive learning techniques for pre-training the networks.
  • NLP for Label Extraction: A crucial step involved using Natural Language Processing (NLP) to automatically extract labels (abnormalities and disease diagnoses) from free-text radiology reports and discharge records. This enabled the creation of a large-scale, automatically annotated dataset.
  • Grad-CAM for Interpretability: To address the "black box" nature of deep learning models, Gradient-weighted Class Activation Mapping (Grad-CAM) was used to visualize which regions of the images the model focused on for its predictions, providing a degree of interpretability.

Datasets

The study utilized large-scale, real-world datasets from two institutions:

  • West China Hospital (WCH): This dataset was used for training and internal validation. It comprised:
  • CT volumes: 191,333 for training/internal testing.
  • CXR images: 125,599 for training/internal testing.
  • Chengdu ShangJin Nanfu Hospital (CSJH): This dataset was used for external validation to assess the model's generalizability and robustness. It comprised:
  • CT volumes: 37,230 for external validation.
  • CXR images: 3,720 for external validation.

The total dataset included over 1.2 million Electronic Health Records (EHRs) from over 430,000 patients, along with 228,563 CT volumes and 129,319 CXR images. The ground truth labels were derived from radiology reports and discharge diagnoses, with a portion manually annotated by medical specialists to ensure quality.

Key Insights and Ideas

  • Mimicking Human Diagnosis: The "abnormality-derived" approach, where disease diagnosis is informed by identified abnormalities, is a key innovation that aims to replicate the clinical reasoning process of radiologists.
  • CT vs. CXR Efficacy: The study provides insights into the relative performance of CT and CXR for diagnosing specific diseases, suggesting that AI assistance can significantly improve the interpretation of CXR images, potentially bridging the diagnostic gap with CT in some cases.
  • Interpretability Matters: The use of Grad-CAM to visualize attention maps is crucial for building trust and understanding in AI diagnostic tools, moving beyond purely black-box predictions.
  • Junior Clinicians Benefit Most: The findings suggest that early-career physicians stand to gain the most from this AI system, potentially leveling the playing field and enhancing their diagnostic capabilities.
  • Robustness Across Institutions: The successful validation on an external dataset from a different institution highlights the system's generalizability, a critical factor for real-world deployment.

Key Results

  • Abnormality Identification: The system achieved an average AUC of 0.856 for CT and 0.841 for CXR.
  • Disease Diagnosis: The system achieved an average AUC of 0.900 for CT and 0.866 for CXR for diagnosing eight major respiratory diseases.
  • Clinical Workflow Integration: When deployed in a preliminary clinical workflow, the system performed on par with senior experts, achieving an AUC of 0.890 and a Cohen's kappa of 0.746-0.877, with a negligible impact on diagnosis time.
  • Interobserver Agreement: The AI system significantly improved the agreement among eight expert clinicians, increasing Cohen's kappa from a median of 0.746 to 0.877 for CT images.
  • External Validation: The system demonstrated robust performance on an external dataset, with mean AUCs of 0.882 for CT and 0.841 for CXR in disease diagnosis.

Limitations and Future Work

  • Annotation Bias: The reliance on NLP for label extraction introduces potential biases, and manual ground-truth validation for such a large dataset is impractical.
  • Generalizability to Other Populations: The study's patient cohort was primarily Asian, limiting its direct generalizability to other international populations. Further validation in diverse populations is recommended.
  • Selection Bias: The selection of specific radiological abnormalities for prediction might introduce bias.
  • Prospective Validation: While the study included retrospective data and a preliminary prospective workflow evaluation, larger-scale prospective studies are needed to fully assess its clinical impact.
  • Future Directions: The authors suggest further research into optimal workflow integration, the use of active learning to continuously improve the model based on physician feedback, and the development of more advanced AI-based algorithms for disease prediction and monitoring.

Summary updated

BibTeX
@article{wang2022development,
  author = {Chengdi Wang and Jiechao Ma and Shu Zhang and Jun Shao and Yanyan Wang and Hong-Yu Zhou and Lujia Song and Jie Zheng and Yizhou Yu and Weimin Li},
  title = {{Development and validation of an abnormality-derived deep-learning diagnostic system for major respiratory diseases}},
  journal = {Npj Digit. Medicine},
  year = {2022},
  doi = {10.1038/s41746-022-00648-z},
}
Figure