COVID-Transformer: Interpretable COVID-19 Detection Using Vision Transformer for Healthcare

Debaditya Shome, T. Kar, Sachi Nandan Mohanty, Prayag Tiwari, Khan Muhammad, Abdullah AlTameem, Yazhou Zhang, Abdul Khader Jilani SaudagarInternational Journal of Environmental Research and Public Health 2021 · DOI

TL;DR

This paper introduces "COVID-Transformer," a Vision Transformer-based deep learning model for detecting COVID-19 from chest X-rays. By aggregating three public datasets into a large, balanced 30K image dataset, they achieve state-of-the-art performance, outperforming traditional CNNs and offering interpretable visualizations for clinical use.

What Problem Does This Paper Address?

The COVID-19 pandemic highlighted a critical need for rapid and accurate diagnostic tools. While RT-PCR is the standard, it has limitations like cost and accessibility. Chest X-rays offer a faster, more accessible alternative, but manual interpretation is time-consuming and prone to errors, especially with the complex and evolving nature of lung involvement in COVID-19. This is particularly challenging in resource-limited settings where expert radiologists are scarce. The paper aims to bridge this gap by developing an automated, accurate, and interpretable system for COVID-19 detection using chest X-rays.

Methods

The core of the proposed solution is a Vision Transformer (ViT) architecture, specifically the ViT L-16 model, pre-trained on ImageNet. The authors adapt this by removing the original MLP prediction block and attaching a custom MLP block consisting of two dense layers with batch normalization and GELU activation. This custom block is designed to process the flattened patches from the Transformer encoder.

For training, they employ a categorical cross-entropy loss for multi-class classification and binary cross-entropy loss for binary classification, incorporating label smoothing to improve generalization. The NovoGrad optimizer is used, noted for its robustness and efficiency. The model is trained on a randomly split dataset (75% train, 15% validation, 10% test).

To enhance interpretability, the researchers utilize Grad-CAM (Gradient-weighted Class Activation Mapping). This technique generates heatmaps that highlight the regions in the X-ray image that are most influential in the model's prediction, allowing radiologists to visualize the affected areas and understand the model's reasoning.

Datasets

The authors created a novel, large-scale dataset by merging data from three existing open-source chest X-ray datasets:

  • El-Shafai et al.
  • Sait et al.
  • Qi et al. (COVID-Ti dataset)

This aggregation resulted in:

  • A 30K image dataset for multi-class classification (COVID-19, Normal, Pneumonia).
  • A 20K image dataset for binary classification (COVID-19 vs. Normal).

Crucially, they ensured these datasets were balanced, with an equal number of images per class. The paper claims this is the largest publicly available, balanced collection of chest X-ray images for COVID-19 detection to date.

Key Insights and Ideas

  • Vision Transformers for Medical Imaging: The paper demonstrates the efficacy of Vision Transformers, a model architecture traditionally dominant in natural language processing and increasingly in computer vision, for the specific task of medical image analysis like COVID-19 detection.
  • The Power of Large, Balanced Datasets: The creation of a 30K image dataset is a significant contribution. The authors highlight that the lack of large, balanced datasets has been a limitation in previous studies, and their aggregated dataset aims to mitigate this, potentially leading to more robust and generalizable models.
  • Interpretability is Key for Clinical Adoption: The integration of Grad-CAM visualizations is a critical insight. It moves beyond a "black box" approach, providing radiologists with visual cues that can build trust and aid in diagnosis, making the AI tool more practical for healthcare settings.
  • ViT's Efficiency: The paper notes that ViT can achieve state-of-the-art performance with potentially less computational resources than CNNs when trained on sufficient data, a valuable characteristic for deployment.

Key Results

The proposed COVID-Transformer model achieved impressive results:

  • Binary Classification (COVID-19 vs. Normal):
    • Accuracy: 98%
    • AUC Score: 99%
  • Multi-class Classification (COVID-19, Normal, Pneumonia):
    • Accuracy: 92%
    • AUC Score: 98%

The model also outperformed several established baseline models (EfficientNetB0, InceptionV3, Resnet50, MobileNetV3, Xception, and DenseNet-121) across various metrics, including accuracy, precision, F1 score, and AUC.

Qualitatively, the Grad-CAM visualizations effectively highlighted affected lung regions, showing more intense activation (yellow/red) for COVID-19 cases compared to pneumonia or normal cases, aligning with clinical understanding of disease severity.

Limitations and Future Work

  • Dataset Limitations: While the authors created a large dataset, they acknowledge that the data is still relatively small compared to general image recognition datasets. They also note that the model sometimes confuses COVID-19 with other types of pneumonia, which is an inherent challenge given their similar radiological appearances.
  • Generalization: The paper mentions that models trained on limited data can lead to improper generalization. While their large dataset aims to address this, further validation on diverse, independent datasets would be beneficial.
  • Future Work: The authors plan to explore another variant of the Vision Transformer architecture to further improve performance, especially with the availability of even larger datasets. They also suggest their approach could be valuable as a second screening method after RT-PCR to reduce false positives/negatives.

Summary updated

Abstract

In the recent pandemic, accurate and rapid testing of patients remained a critical task in the diagnosis and control of COVID-19 disease spread in the healthcare industry. Because of the sudden increase in cases, most countries have faced scarcity and a low rate of testing. Chest X-rays have been shown in the literature to be a potential source of testing for COVID-19 patients, but manually checking X-ray reports is time-consuming and error-prone. Considering these limitations and the advancements in data science, we proposed a Vision Transformer-based deep learning pipeline for COVID-19 detection from chest X-ray-based imaging. Due to the lack of large data sets, we collected data from three open-source data sets of chest X-ray images and aggregated them to form a 30 K image data set, which is the largest publicly available collection of chest X-ray images in this domain to our knowledge. Our proposed transformer model effectively differentiates COVID-19 from normal chest X-rays with an accuracy of 98% along with an AUC score of 99% in the binary classification task. It distinguishes COVID-19, normal, and pneumonia patient's X-rays with an accuracy of 92% and AUC score of 98% in the Multi-class classification task. For evaluation on our data set, we fine-tuned some of the widely used models in literature, namely, EfficientNetB0, InceptionV3, Resnet50, MobileNetV3, Xception, and DenseNet-121, as baselines. Our proposed transformer model outperformed them in terms of all metrics. In addition, a Grad-CAM based visualization is created which makes our approach interpretable by radiologists and can be used to monitor the progression of the disease in the affected lungs, assisting healthcare.
BibTeX
@article{shome2021covid,
  author = {Debaditya Shome and T. Kar and Sachi Nandan Mohanty and Prayag Tiwari and Khan Muhammad and Abdullah AlTameem and Yazhou Zhang and Abdul Khader Jilani Saudagar},
  title = {{COVID-Transformer: Interpretable COVID-19 Detection Using Vision Transformer for Healthcare}},
  journal = {International Journal of Environmental Research and Public Health},
  year = {2021},
  doi = {10.3390/ijerph182111086},
}
Figure