Generalized Deep Learning Models for COVID-19 Detection with Transfer and Continual Learning

Richard Annan, Hong Qin, Letu QinggeEPiC 2024 · DOI

TL;DR

Researchers have developed a more "generalizable" approach to COVID-19 detection from CT scans by combining transfer learning with Continual Learning. By implementing a modified Elastic Weight Consolidation (EWC) technique, their models can adapt to new, diverse datasets from different global regions without "forgetting" the patterns they learned from their original training data.

What Problem Does This Paper Address?

While deep learning models have shown high accuracy in identifying COVID-19 pneumonia in CT scans, they suffer from a major "brittleness" problem: lack of generalization. A model trained on data from one hospital often fails miserably when tested on data from another.

The standard fix is transfer learning—taking a pre-trained model and fine-tuning it on a new dataset. However, this triggers catastrophic forgetting, where the model's performance on the original dataset plummets as it overwrites old weights to accommodate new information. In a clinical setting, we need "evergreen" models that can learn from new virus strains or different imaging hardware without losing their previous diagnostic capabilities.

Methods

The authors evaluated three distinct architectures to see which handled generalization best:

  1. COVID-CNN: A custom Convolutional Neural Network tailored for grayscale CT images.
  2. Capsule Network (CapsNet): A model that uses "capsules" and dynamic routing to better understand hierarchical relationships between image features (like the spatial orientation of lung lesions).
  3. Vision Transformer (ViT): A modern architecture that uses self-attention mechanisms to capture global relationships across image patches.

The Innovation: Modified EWC To solve catastrophic forgetting, the authors applied a modified version of Elastic Weight Consolidation (EWC). Think of EWC as a "protective anchor" for the model's brain. It adds a regularization term to the loss function that penalizes changes to weights that were crucial for the previous task. By slowing down the updates to these "important" weights, the model can learn from a new dataset (Task B) while retaining the knowledge of the original dataset (Task A).

Datasets

The study utilized two primary aggregated datasets:

  • dataset_mod_dev (Development): 4,649 images sourced from radiologist centers in Brazil and Iran. This served as the "original" knowledge base.
  • dataset_mod_gen (Generalization): A larger, more diverse set of 14,486 images from Russia, China, Italy, Turkey, and Iran. This was used to test how well the models could adapt to global variations.

Key Insights and Ideas

  • The Generalization Gap is Real: Without EWC, a CNN trained on the development set achieved 97.85% accuracy but dropped to a dismal 47.96% when tested on the generalization set.
  • Transfer Learning is a Double-Edged Sword: Fine-tuning the CNN on the new data boosted its accuracy on that new data to 93.75%, but its performance on the original data fell to 50.43%. It effectively "replaced" its old memory with new memory.
  • Simplicity Wins: Interestingly, the custom COVID-CNN outperformed the more complex Vision Transformer and CapsNet in terms of consistent knowledge retention. The authors attribute this to its parameter efficiency and less complex architecture, which makes it more compatible with effective regularization.
  • The Balancing Act: The hyperparameter $\lambda$ (lambda) is the "knob" that controls the trade-off. A value of 2.0 was found to be the "sweet spot" for balancing the acquisition of new knowledge with the retention of the old.

Key Results

  • Superior Retention: The COVID-CNN with EWC achieved an 82.26% knowledge retention rate on the original dataset while maintaining a solid 79.33% accuracy on the new, unseen global dataset.
  • Comparison with SOTA: The proposed models significantly outperformed existing state-of-the-art transfer learning models (like Deep-COVID and Deep COVID DeteCT), which showed massive performance drops (down to ~28-41% accuracy) when faced with the generalization dataset.
  • Stability: While CapsNet performed slightly better on the new data (80.35%), the COVID-CNN was deemed the most reliable overall due to its superior balance and consistency across both datasets.

Limitations and Future Work

The authors acknowledge that while their modified EWC is faster and more accurate than the original version, continual learning is still an evolving field.

  • Computational Intensity: Even simplified EWC requires more overhead than standard training.
  • Hyperparameter Sensitivity: Finding the optimal $\lambda$ currently requires trial and error; automating this process would be a significant step forward.
  • Scaling Up: Future work should involve testing these models on even larger, multi-modal datasets (combining CT scans with clinical metadata) to further improve diagnostic robustness.

Summary updated

Abstract

Deep learning has achieved great success for detecting COVID-19 from CT scan images. However, there is lack of generalization ability for the existing models. For example, one model with a higher prediction accuracy developed on one dataset cannot be used to pre- dict on another dataset. Thus, developing a robust deep learning model that has a great generalization ability is a significant need. In this paper, we first apply three deep learning models, namely convolutional neural network (CNN), capsule neural network (CapsNet) and vision transformer (ViT) and test their generalization abilities. Then, we develop and hypertune the models based on transfer learning to generalize the model performance on new datasets. However, the transfer learning technique always has the catastrophic forgetting issue which lead to lower prediction accuracy on its original training dataset. Lastly, we will apply continual learning based on modified elastic weight consolidation (EWC) regularization technique to address the catastrophic forgetting issue and improve the models’ prediction accuracy on both new and original training datasets. Our results on cross-data validation show that our proposed models not only achieve better prediction accuracy of up to 97.85% compared with the existing state-of-the-art models, but also the proposed models with EWC show great generalization ability and retain the higher prediction accuracy on both new dataset and the training dataset. Extensive experiments show that our proposed COVID-CNN model with EWC outperforms ViT and CapsNet with an impressive 82.26% knowledge retention rate on the original training dataset. Our developed code can be found from https://github.com/astonish24/-QinggeLab BICOB24.
BibTeX
@inproceedings{annan2024generalized,
  author = {Richard Annan and Hong Qin and Letu Qingge},
  title = {{Generalized Deep Learning Models for COVID-19 Detection with Transfer and Continual Learning}},
  booktitle = {EPiC},
  year = {2024},
  doi = {10.29007/952w},
}
Figure