Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans

Xin He, Shihao Wang, Shaohuai Shi, Xiaowen Chu, Jiangping Tang, Xin Liu, Chenggang Yan, Jiyong Zhang, Guiguang DingmedRxiv 2020 · DOI

Abstract

COVID-19 pandemic has spread all over the world for months. As its transmissibility and high pathogenicity seriously threaten people’s lives, the accurate and fast detection of the COVID-19 infection is crucial. Although many recent studies have shown that deep learning based solutions can help detect COVID-19 based on chest CT scans, there lacks a consistent and systematic comparison and evaluation on these techniques. In this paper, we first build a clean and segmented CT dataset called Clean-CC-CCII by fixing the errors and removing some noises in a large CT scan dataset CC-CCII with three classes: novel coronavirus pneumonia (NCP), common pneumonia (CP), and normal controls (Normal). After cleaning, our dataset consists of a total of 340,190 slices of 3,993 scans from 2,698 patients. Then we benchmark and compare the performance of a series of state-of-the-art (SOTA) 3D and 2D convolutional neural networks (CNNs). The results show that 3D CNNs outperform 2D CNNs in general. With extensive effort of hyperparameter tuning, we find that the 3D CNN model DenseNet3D121 achieves the highest accuracy of 88.63% (F1-score is 88.14% and AUC is 0.940), and another 3D CNN model ResNet3D34 achieves the best AUC of 0.959 (accuracy is 87.83% and F1-score is 86.04%). We further demonstrate that the mixup data augmentation technique can largely improve the model performance. At last, we design an automated deep learning methodology to generate a lightweight deep learning model MNas3DNet41 that achieves an accuracy of 87.14%, F1-score of 87.25%, and AUC of 0.957, which are on par with the best models made by AI experts. The automated deep learning design is a promising methodology that can help health-care professionals develop effective deep learning models using their private data sets.

Introduction

The rapid spread of COVID-19 has spurred numerous deep learning (DL) initiatives for automated diagnosis. However, the field has suffered from a lack of standardization due to varying datasets, preprocessing steps, and inconsistent evaluation metrics. Furthermore, many public datasets are "noisy," containing errors like disordered slices or non-lung images. This research addresses these gaps by establishing a clean benchmark and systematically evaluating state-of-the-art (SOTA) architectures to identify the most effective strategies for automated diagnosis.

The Clean-CC-CCII Dataset

The authors manually refined the large-scale CC-CCII dataset to create Clean-CC-CCII. This benchmark includes three distinct classes: Novel Coronavirus Pneumonia (NCP), Common Pneumonia (CP), and Normal controls.

Representative CT scan slices from the Clean-CC-CCII dataset for (a) NCP, (b) CP, and (c) Normal controls.

The cleaning process involved fixing slice orders, removing irrelevant anatomical images, and implementing K-Means-based lung segmentation to isolate relevant regions. The final dataset comprises 340,190 slices from 3,993 scans of 2,698 patients, making it one of the most robust resources for COVID-19 CT research.

Methodology

The study employed a systematic three-stage approach to evaluate and optimize detection models.

The proposed pipeline for COVID-19 detection, including slice sampling, preprocessing, and 3D/2D CNN classification.
  1. Data Refinement: Manual cleaning and automated segmentation were used to ensure high-quality input.
  2. Systematic Benchmarking: 17 CNN architectures were compared, analyzing the impact of model dimensionality (2D vs. 3D), depth, and input channel strategies (greyscale vs. RGB).
  3. Automated Model Design: Using Neural Architecture Search (NAS), the researchers developed a 3D version of the MobileNetV2 search space. A random search strategy identified MNas3DNet, a lightweight 3D architecture optimized for both accuracy and computational efficiency.

Key Insights and Results

The benchmarking process yielded several critical findings for medical AI development:

  • Superiority of 3D Architectures: 3D CNNs consistently outperformed 2D models by capturing volumetric features and spatial relationships across multiple CT slices.
  • Optimal Model Depth: Deeper models did not always yield better results. For instance, ResNet3D34 achieved a superior AUC (0.959) compared to its 101-layer and 200-layer counterparts.
  • Efficiency of Greyscale: Using single-channel greyscale images often resulted in higher AUC scores than "fake" 3-channel RGB images, suggesting that redundant channels can complicate training.
  • Data Augmentation: The "mixup" technique—linearly combining image pairs and labels—improved accuracy by up to 4.27%.
  • Top Performers: DenseNet3D121 led in accuracy (88.63%), while the NAS-generated MNas3DNet41 achieved a competitive 87.14% accuracy with a significantly smaller footprint (22.91 MB), making it ideal for resource-constrained clinical settings.

Conclusion and Future Work

While Clean-CC-CCII provides a strong foundation, expanding the dataset with more diverse global data is essential for generalizability. Future research will shift from simple classification to semantic segmentation, allowing models to delineate infected areas to help clinicians track disease progression and severity. Additionally, the team plans to explore more advanced NAS strategies beyond the random search used in this study.

Summary updated

Abstract

Abstract COVID-19 pandemic has spread all over the world for months. As its transmissibility and high pathogenicity seriously threaten people’s lives, the accurate and fast detection of the COVID-19 infection is crucial. Although many recent studies have shown that deep learning based solutions can help detect COVID-19 based on chest CT scans, there lacks a consistent and systematic comparison and evaluation on these techniques. In this paper, we first build a clean and segmented CT dataset called Clean-CC-CCII by fixing the errors and removing some noises in a large CT scan dataset CC-CCII with three classes: novel coronavirus pneumonia (NCP), common pneumonia (CP), and normal controls (Normal). After cleaning, our dataset consists of a total of 340,190 slices of 3,993 scans from 2,698 patients. Then we benchmark and compare the performance of a series of state-of-the-art (SOTA) 3D and 2D convolutional neural networks (CNNs). The results show that 3D CNNs outperform 2D CNNs in general. With extensive effort of hyperparameter tuning, we find that the 3D CNN model DenseNet3D121 achieves the highest accuracy of 88.63% (F1-score is 88.14% and AUC is 0.940), and another 3D CNN model ResNet3D34 achieves the best AUC of 0.959 (accuracy is 87.83% and F1-score is 86.04%). We further demonstrate that the mixup data augmentation technique can largely improve the model performance. At last, we design an automated deep learning methodology to generate a lightweight deep learning model MNas3DNet41 that achieves an accuracy of 87.14%, F1-score of 87.25%, and AUC of 0.957, which are on par with the best models made by AI experts. The automated deep learning design is a promising methodology that can help health-care professionals develop effective deep learning models using their private data sets. Our Clean-CC-CCII dataset and source code are available at: https://github.com/HKBU-HPML/HKBU_HPML_COVID-19 .
BibTeX
@misc{he2020benchmarking,
  author = {Xin He and Shihao Wang and Shaohuai Shi and Xiaowen Chu and Jiangping Tang and Xin Liu and Chenggang Yan and Jiyong Zhang and Guiguang Ding},
  title = {{Benchmarking Deep Learning Models and Automated Model Design for COVID-19 Detection with Chest CT Scans}},
  year = {2020},
  doi = {10.1101/2020.06.08.20125963},
  howpublished = {medRxiv},
}
Figure