Robust Automated Framework for COVID-19 Disease Identification from a Multicenter Dataset of Chest CT Scans

Shahin Heidarian, Parnian Afshar, Nastaran Enshaei, Farnoosh Naderkhani, Moezedin Javad Rafiee, Anastasia Oikonomou, Akbar Shafiee, Faranak Babaki Fard, Konstantinos N. Plataniotis, Arash MohammadiarXiv 2021 · OpenReview

TL;DR

This paper introduces a robust, two-stage deep learning framework for identifying COVID-19, Community-Acquired Pneumonia (CAP), and normal cases from chest CT scans. Crucially, it demonstrates how this framework can be effectively updated using unlabeled data from new, heterogeneous datasets, significantly improving its generalization capabilities and robustness.

What Problem Does This Paper Address?

The COVID-19 pandemic highlighted the urgent need for accurate and rapid diagnostic tools. Chest CT scans are highly sensitive for detecting COVID-19, but their interpretation is challenging due to overlapping features with other pneumonias (like CAP) and the inherent variability in CT imaging. This variability arises from different scanners, imaging protocols, and radiation doses used across various medical centers. Standard deep learning models trained on data from a single source often perform poorly when applied to data from different sources (a phenomenon known as "domain shift"). This paper tackles the critical problem of building a deep learning model that is not only accurate but also robust and adaptable to these real-world variations in CT imaging data, making it more clinically applicable.

Methods

The core of the proposed solution is a two-stage framework built upon Capsule Networks (CapsNets).

  • Stage 1: Infection Identification: This stage acts as a pre-filter. It takes a CT scan, extracts the lung areas (using a pre-trained U-Net segmentation model), and then uses a CapsNet to identify slices that show evidence of infection (regardless of whether it's COVID-19 or CAP). This stage is designed to be robust even with imbalanced data, employing a weighted loss function. A key innovation here is a thresholding mechanism: if less than 3% of slices in a scan are flagged as infectious, the entire scan is classified as "Normal." This significantly reduces the need to analyze every slice and helps in early identification of normal cases.
  • Stage 2: Three-Way Classification: Slices identified as potentially infectious from Stage 1 are then fed into a second CapsNet. This stage classifies them into one of the three target classes: COVID-19, CAP, or Normal. Similar to Stage 1, it uses a weighted loss function to handle class imbalance, with specific weights adjusted to prioritize minority classes.
  • Unsupervised Enhancement: This is a critical component for robustness. When faced with new, unseen CT scans from different centers or with different imaging parameters, the model can be updated without requiring new labels. The framework identifies "confident" predictions from the test data (based on high probability scores) and uses these confident slices to retrain the Stage 2 model. This "unsupervised learning" approach allows the model to adapt to data shifts.
  • Ensemble Architecture: To further enhance performance and robustness, the predictions from multiple versions of the model (each enhanced on different test sets) are aggregated using an ensemble approach.

Datasets

  • Training Dataset: An in-house dataset called "COVID-CT-MD" was used. It consists of 171 COVID-19, 60 CAP, and 76 Normal cases, all acquired from a single imaging center with a consistent scanning protocol and standard radiation dose.
  • Test Datasets: Four distinct test sets were collected retrospectively, designed to represent various real-world challenges:
  • Test Set 1: Low and Ultra-Low dose CT scans.
  • Test Set 2: Scans from a different imaging center with varied scanning parameters, including some cases with a history of cardiovascular disease/surgeries.
  • Test Set 3: Scans acquired using the same protocol as the training set but from different patients.
  • Test Set 4: A combination of new scans from both centers, using similar acquisition settings. The SPGC-COVID dataset, which comprises these four test sets, was also introduced and made publicly available.

Key Insights and Ideas

  • Capsule Networks for Robustness: The paper champions CapsNets for their superior discriminative capability, especially with limited annotated data, and their ability to capture spatial relationships, which is crucial for medical imaging.
  • Unsupervised Enhancement is Key: The most significant insight is the effectiveness of the unsupervised enhancement approach. It demonstrates that a model can learn from new, unlabeled data to adapt to domain shifts, a crucial step towards clinically deployable AI.
  • Thresholding for Normal Cases: The simple yet effective 3% slice-based threshold for identifying normal cases is a clever way to reduce computational load and improve efficiency without sacrificing accuracy for normal cases.
  • Ensemble for Generalization: Aggregating predictions from models enhanced on different test sets provides a powerful way to achieve robust performance across diverse data sources.

Key Results

The proposed framework achieved impressive results:

  • Overall Accuracy: 96.15% on the combined test sets.
  • COVID-19 Sensitivity: 96.08%.
  • CAP Sensitivity: 92.86%.
  • Normal Sensitivity: 98.04%.
  • AUC: 0.992.

The unsupervised enhancement approach significantly improved performance, especially on heterogeneous test sets. For instance, the benchmark model (without enhancement) achieved 93.85% accuracy, while the enhanced model reached 96.15%. The paper also shows that the model enhanced on one test set can improve performance on others, highlighting the generalizability of the enhancement strategy.

Limitations and Future Work

  • Dataset Diversity: While the SPGC-COVID dataset is a good step, the authors acknowledge the challenge of obtaining truly diverse datasets from many countries and imaging modalities. Future work will focus on expanding dataset diversity to further investigate generalizability.
  • Adaptive Thresholding: The fixed 3% threshold for normal cases might be too simplistic for scans with very few slices or unusual slice thicknesses. The authors suggest exploring adaptive thresholds for more robust normal case identification.
  • Advanced Data Selection: The paper mentions the potential for more advanced techniques like Active Learning to select more diverse and informative samples for retraining, which could further boost performance.
  • Domain Invariant Representations: While the current approach is effective, future work could explore techniques like Generative Adversarial Networks (GANs) to create domain-invariant image representations, which might be beneficial for even greater robustness across different imaging modalities.

Summary updated

Abstract

The objective of this study is to develop a robust deep learning-based framework to distinguish COVID-19, Community-Acquired Pneumonia (CAP), and Normal cases based on chest CT scans acquired in different imaging centers using various protocols, and radiation doses. We showed that while our proposed model is trained on a relatively small dataset acquired from only one imaging center using a specific scanning protocol, the model performs well on heterogeneous test sets obtained by multiple scanners using different technical parameters. We also showed that the model can be updated via an unsupervised approach to cope with the data shift between the train and test sets and enhance the robustness of the model upon receiving a new external dataset from a different center. We adopted an ensemble architecture to aggregate the predictions from multiple versions of the model. For initial training and development purposes, an in-house dataset of 171 COVID-19, 60 CAP, and 76 Normal cases was used, which contained volumetric CT scans acquired from one imaging center using a constant standard radiation dose scanning protocol. To evaluate the model, we collected four different test sets retrospectively to investigate the effects of the shifts in the data characteristics on the model's performance. Among the test cases, there were CT scans with similar characteristics as the train set as well as noisy low-dose and ultra-low dose CT scans. In addition, some test CT scans were obtained from patients with a history of cardiovascular diseases or surgeries. The entire test dataset used in this study contained 51 COVID-19, 28 CAP, and 51 Normal cases. Experimental results indicate that our proposed framework performs well on all test sets achieving total accuracy of 96.15% (95%CI: [91.25-98.74]), COVID-19 sensitivity of 96.08% (95%CI: [86.54-99.5]), CAP sensitivity of 92.86% (95%CI: [76.50-99.19]).
BibTeX
@misc{heidarian2021robust,
  author = {Shahin Heidarian and Parnian Afshar and Nastaran Enshaei and Farnoosh Naderkhani and Moezedin Javad Rafiee and Anastasia Oikonomou and Akbar Shafiee and Faranak Babaki Fard and Konstantinos N. Plataniotis and Arash Mohammadi},
  title = {{Robust Automated Framework for COVID-19 Disease Identification from a Multicenter Dataset of Chest CT Scans}},
  year = {2021},
  eprint = {2109.09241},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.09241},
}
Figure