Self-supervised Model Based on Masked Autoencoders Advance CT Scans Classification

Jiashu Xu, Sergii G. StirenkoarXiv 2022 · DOI · OpenReview

TL;DR

This paper introduces a self-supervised learning approach using Masked Autoencoders (MAE) for classifying medical CT scans, specifically for COVID-19 detection. By pre-training on large, unlabeled datasets and then fine-tuning on smaller, labeled medical datasets, their method achieves performance comparable to supervised learning while mitigating the need for extensive manual annotation and reducing overfitting.

What Problem Does This Paper Address?

The field of medical image analysis, particularly for tasks like COVID-19 diagnosis from CT scans, faces a significant hurdle: the scarcity of large, publicly available, and expertly annotated datasets. Obtaining such annotations is costly and time-consuming, requiring specialized medical professionals. This limitation hinders the widespread adoption and effectiveness of deep learning models, which typically thrive on vast amounts of labeled data. While supervised learning has shown promise, its reliance on labeled data makes it susceptible to overfitting when applied to smaller medical datasets. This paper aims to bridge this gap by leveraging self-supervised learning to learn robust feature representations from unlabeled data, thereby reducing the dependency on manual annotations and improving generalization.

Methods

The core of this work is the adaptation of the Masked Autoencoder (MAE) architecture, originally designed for natural images, to the domain of medical CT scans. The MAE approach involves an asymmetric encoder-decoder structure.

  • Pre-training (Self-Supervised): The encoder, based on the Vision Transformer (ViT) architecture, takes an image where a significant portion (up to 75%) of patches are randomly masked. The decoder's task is to reconstruct these missing patches. This process allows the encoder to learn rich, contextual representations of the image data without any explicit labels. The authors leverage a pre-trained MAE model on the ImageNet dataset as a starting point.
  • Fine-tuning (Transfer Learning): After the self-supervised pre-training, the encoder is then fine-tuned on specific downstream tasks, such as COVID-19 classification, using labeled CT scan datasets. This transfer learning step adapts the general features learned during pre-training to the specific medical imaging task.

The authors highlight that their MAE approach is distinct from some other self-supervised methods like contrastive learning (e.g., MOCO) in that it focuses on reconstruction rather than direct comparison of samples, potentially offering improved efficiency.

Datasets

The paper utilizes several datasets for their experiments:

  • COVID-CTset: A large dataset containing over 60,000 CT images, used for the self-supervised pre-training stage.
  • COVID-CT-Dataset: A smaller dataset with 397 COVID-19 negative and 349 COVID-19 positive samples, used for downstream task evaluation.
  • SARS-CoV-2 CT Scan Dataset: A dataset from Kaggle with approximately 2482 samples, also used for downstream task evaluation.
  • ImageNet: A large-scale dataset used for initial pre-training of the MAE model before fine-tuning on medical data.

Key Insights and Ideas

  • MAE's Effectiveness in Medical Imaging: The study demonstrates that MAE, a technique proven effective in natural image domains, can be successfully adapted for medical CT scan classification, achieving competitive results.
  • Pre-training Domain Matters (but not always as expected): While pre-training on a domain closer to the target (like COVID-CTset) might seem intuitive, the experiments suggest that pre-training on a large, diverse, and unrelated domain like ImageNet can be equally or even more effective for learning generalizable features. This is attributed to the need for large-scale data to learn robust representations.
  • Adding Target Domain Pre-training Boosts Performance: Incorporating an additional pre-training step on the target medical domain (even without labels) after initial ImageNet pre-training further enhances performance, indicating the value of domain-specific feature learning.
  • Self-Supervised Learning as a Viable Alternative: The research strongly supports self-supervised learning as a powerful alternative to supervised learning in medical imaging, offering comparable performance with significantly reduced reliance on labeled data.

Key Results

  • The MAE-based self-supervised learning method achieved performance comparable to supervised learning methods on both the COVID-CT and SARS-CoV-2 datasets.
  • In some comparisons, the MAE model even outperformed traditional backbone networks like VGG16, ResNet101, and DenseNet121 when used with supervised pre-training.
  • The training curves for the MAE-based models were observed to be smoother, indicating better stability and generalization performance compared to randomly initialized models.
  • Ablation studies showed that pre-training on ImageNet was crucial, and adding a pre-training step on the target domain (COVID-CT or SARS-CoV-2) further improved results.

Limitations and Future Work

  • Domain Discrepancy: The authors acknowledge the significant domain discrepancy between ImageNet and CT scans. While their method shows effectiveness, further research could explore more domain-adaptive pre-training strategies.
  • Computational Resources: While MAE is presented as more efficient than contrastive methods, self-supervised learning, in general, can still be computationally intensive, especially during the pre-training phase.
  • Future Work: The paper suggests that this method can help improve the efficiency of medical image diagnosis and provide prior experience for researchers and medical image analysts. Future work could explore applying this to other medical imaging modalities or different diagnostic tasks. The authors also mention their code will be open-sourced, facilitating further research.

Summary updated

Abstract

The coronavirus pandemic has been going on since the year 2019, and the trend is still not abating. Therefore, it is particularly important to classify medical CT scans to assist in medical diagnosis. At present, Supervised Deep Learning algorithms have made a great success in the classification task of medical CT scans, but medical image datasets often require professional image annotation, and many research datasets are not publicly available. To solve this problem, this paper is inspired by the self-supervised learning algorithm MAE and uses the MAE model pre-trained on ImageNet to perform transfer learning on CT Scans dataset. This method improves the generalization performance of the model and avoids the risk of overfitting on small datasets. Through extensive experiments on the COVID-CT dataset and the SARS-CoV-2 dataset, we compare the SSL-based method in this paper with other state-of-the-art supervised learning-based pretraining methods. Experimental results show that our method improves the generalization performance of the model more effectively and avoids the risk of overfitting on small datasets. The model achieved almost the same accuracy as supervised learning on both test datasets. Finally, ablation experiments aim to fully demonstrate the effectiveness of our method and how it works.
BibTeX
@misc{xu2022self,
  author = {Jiashu Xu and Sergii G. Stirenko},
  title = {{Self-supervised Model Based on Masked Autoencoders Advance CT Scans Classification}},
  year = {2022},
  eprint = {2210.05073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.05073},
}
Figure