Development and evaluation of an artificial intelligence system for COVID-19 diagnosis

Cheng Jin, Weixiang Chen, Yukun Cao, Zhanwei Xu, Zimeng Tan, Xin Zhang, Lei Deng, Chuansheng Zheng, Jie Zhou, Heshui Shi, Jianjiang FengNature Communications 2020 · DOI

Abstract

Early detection of COVID-19 based on chest CT enables timely treatment of patients and helps control the spread of the disease. We proposed an artificial intelligence (AI) system for rapid COVID-19 detection and performed extensive statistical analysis of CTs of COVID-19 based on the AI system. We developed and evaluated our system on a large dataset with more than 10 thousand CT volumes from COVID-19, influenza-A/B, non-viral community acquired pneumonia (CAP) and non-pneumonia subjects. In such a difficult multi-class diagnosis task, our deep convolutional neural network-based system is able to achieve an area under the receiver operating characteristic curve (AUC) of 97.81% for multi-way classification on test cohort of 3,199 scans, AUC of 92.99% and 93.25% on two publicly available datasets, CC-CCII and MosMedData respectively. In a reader study involving five radiologists, the AI system outperforms all of radiologists in more challenging tasks at a speed of two orders of magnitude above them. Diagnosis performance of chest x-ray (CXR) is compared to that of CT. Detailed interpretation of deep network is also performed to relate system outputs with CT presentations.

Introduction

Early in the COVID-19 pandemic, the "gold standard" RT-PCR testing faced significant hurdles, including low sensitivity in patients with low viral loads and severe supply shortages. While chest CT scans offer a high-sensitivity alternative for detecting lung lesions, they present a different bottleneck: a single CT volume contains hundreds of slices, making manual review by radiologists time-consuming. Furthermore, COVID-19 shares many visual characteristics with other respiratory illnesses, such as influenza and non-viral community-acquired pneumonia (CAP). Distinguishing between these "look-alike" diseases is a high-stakes challenge that requires significant expertise and time—resources that are often stretched thin during an outbreak. This study presents a deep learning-based AI system capable of diagnosing COVID-19 from chest CT scans with human-surpassing accuracy and remarkable speed.

Methods and Materials

The researchers developed a multi-stage AI pipeline designed for both accuracy and interpretability. The process begins with lung segmentation using a U-Net architecture to isolate lung regions and remove background noise. Following this, individual 2D slices are classified using a ResNet-152 backbone into four categories: COVID-19, Influenza, CAP, or non-pneumonia. A custom fusion module then aggregates scores from all slices in a volume to produce a final patient-level diagnosis.

Workflow of the AI system development and evaluation process.

To ensure generalizability, the system was trained and validated on a massive, diverse dataset:

  • Primary Dataset: 11,356 CT scans from 9,025 subjects across three centers in Wuhan, China.
  • Public Benchmarks: Validation against the CC-CCII and MosMedData (Russian) datasets.
  • Interpretability: The team employed Guided Grad-CAM to visualize AI focus areas and used radiomics to correlate mathematical findings with clinical phenotypes like "ground-glass opacities."

Results

The AI system achieved an Area Under the Curve (AUC) of 97.81% for multi-way classification on the main test cohort. It maintained high performance on external datasets, with AUCs of 92.99% (CC-CCII) and 93.25% (MosMedData). For COVID-19 specifically, the system reached 87.03% sensitivity and 96.60% specificity.

ROC curves showing the AI system's performance across different diagnostic categories on the test cohort.

In a head-to-head reader study, the AI outperformed five board-certified radiologists in the most difficult tasks, such as distinguishing COVID-19 from Influenza. Crucially, the AI processed a scan in roughly 2.73 seconds, compared to an average of 6.5 minutes for a radiologist. This represents a 140x speedup, which is transformative for triage in overwhelmed hospitals. Additionally, the study quantitatively confirmed that CT is significantly more effective than chest X-ray (CXR) for COVID-19 diagnosis.

Discussion and Conclusion

By utilizing radiomics, the authors demonstrated that the AI focuses on specific features like "pleural parallel signs" and "vascular thickening," aligning with the pathological understanding of COVID-19. While the system shows immense promise, future work should focus on increasing subtype diversity in training to further reduce false positives and integrating non-imaging clinical data (e.g., symptoms, blood work) to provide a more holistic severity score. The current system provides a robust, rapid tool for clinical triage, significantly reducing the burden on radiologists during pandemic surges.

Summary updated

Abstract

Abstract Early detection of COVID-19 based on chest CT enables timely treatment of patients and helps control the spread of the disease. We proposed an artificial intelligence (AI) system for rapid COVID-19 detection and performed extensive statistical analysis of CTs of COVID-19 based on the AI system. We developed and evaluated our system on a large dataset with more than 10 thousand CT volumes from COVID-19, influenza-A/B, non-viral community acquired pneumonia (CAP) and non-pneumonia subjects. In such a difficult multi-class diagnosis task, our deep convolutional neural network-based system is able to achieve an area under the receiver operating characteristic curve (AUC) of 97.81% for multi-way classification on test cohort of 3,199 scans, AUC of 92.99% and 93.25% on two publicly available datasets, CC-CCII and MosMedData respectively. In a reader study involving five radiologists, the AI system outperforms all of radiologists in more challenging tasks at a speed of two orders of magnitude above them. Diagnosis performance of chest x-ray (CXR) is compared to that of CT. Detailed interpretation of deep network is also performed to relate system outputs with CT presentations. The code is available at https://github.com/ChenWWWeixiang/diagnosis_covid19 .
BibTeX
@inproceedings{jin2020development,
  author = {Cheng Jin and Weixiang Chen and Yukun Cao and Zhanwei Xu and Zimeng Tan and Xin Zhang and Lei Deng and Chuansheng Zheng and Jie Zhou and Heshui Shi and Jianjiang Feng},
  title = {{Development and evaluation of an artificial intelligence system for COVID-19 diagnosis}},
  booktitle = {Nature Communications},
  year = {2020},
  doi = {10.1038/s41467-020-18685-1},
}
Figure