Semi-supervised COVID-19 CT image segmentation using deep generative models

Judah Zammit, Daryl L. X. Fung, Qian Liu, Carson Kai-Sang Leung, Pingzhao Hu

Introduction

The rapid diagnosis and monitoring of COVID-19 rely heavily on medical imaging, particularly Computed Tomography (CT) scans. Accurate image segmentation—the process of classifying every pixel in an image—is essential for quantifying lung damage, such as ground-glass opacity (GGO) and consolidation. However, the development of robust deep learning models for this task is hindered by a significant data bottleneck. While raw CT data is abundant, obtaining high-quality, pixel-level annotations requires expert radiologists to perform labor-intensive manual labeling.

To address this challenge, researchers have proposed StitchNet, a novel semi-supervised learning (SSL) framework. By leveraging both a small set of labeled images and a vast amount of unlabeled data, StitchNet aims to achieve high-performance segmentation without the prohibitive cost of exhaustive manual annotation.

The Challenge of Medical Image Segmentation

In computer vision, semi-supervised learning has seen great success in image classification. However, extending these techniques to segmentation is difficult. Unlike classification, where a single label is assigned to an entire image, segmentation requires predicting a label for every pixel while maintaining spatial consistency. The number of possible pixel combinations is astronomical, making it harder for models to generalize from limited labeled data.

Figure 6: A CT scan exhibiting diffuse ground-glass opacities (GGO), a hallmark of COVID-19 lung involvement.

Methodology: SVAE and StitchNet

The core of the proposed solution lies in two architectural innovations: the Shared Variational Autoencoder (SVAE) and the StitchNet framework.

Shared Variational Autoencoder (SVAE)

Building upon the Ladder VAE (LVAE) architecture, the SVAE utilizes a five-layer hierarchy of latent variables to capture complex image features. A key distinction of the SVAE is its weight-sharing mechanism between the inference (encoder) and generative (decoder) networks. The researchers hypothesize that because encoding (extracting features) and decoding (reconstructing images) are essentially inverse operations, sharing weights can improve model efficiency and feature representation.

The StitchNet Framework

StitchNet applies the SVAE to the segmentation task through a unique "stitching" process. Instead of directly outputting a final mask, the model generates four distinct "stylistic" versions of the input CT scan:

  1. Healthy Lung Field: A representation of what the lung would look like without pathology.
  2. Ground-Glass Opacity (GGO): A style capturing the hazy, increased-attenuation areas.
  3. Consolidation: A style representing denser, more opaque lung tissue.
  4. Background: The non-lung areas of the scan.

The model predicts a segmentation mask and then "stitches" the original image back together by selecting pixels from these four styles based on the mask's predictions. This reconstruction task acts as a form of self-supervision; for the model to accurately reconstruct the raw CT scan, it must produce a physically and anatomically meaningful segmentation mask.

Experimental Setup and Dataset

The study utilized the China Consortium of Chest CT Image Investigation (CC-CCII) dataset, which contains over 440,000 CT images from 2,778 patients. For the supervised component, a subset of 750 images was meticulously annotated by five senior radiologists, providing ground-truth masks for healthy lung tissue, GGO, and consolidation.

Figure 4: An example of a segmented CT scan, highlighting the complex task of identifying specific lesion boundaries within the lung field.

Key Results and Insights

The performance of StitchNet was compared against established fully supervised architectures, including U-Net (with a MobileNetV2 backbone) and SegNet.

  • Performance Parity: Despite using significantly fewer labels, StitchNet achieved an overall Intersection-over-Union (IoU) of 0.554. This is remarkably close to the 0.593 IoU achieved by the fully supervised U-Net, demonstrating the effectiveness of the semi-supervised approach.
  • Enhanced Consistency: One of the most significant findings was that StitchNet exhibited a lower standard deviation in its performance metrics compared to supervised models. This suggests that training on unlabeled data provides a regularizing effect, leading to more stable and reliable predictions across diverse patient scans.
  • Conservative Clinical Profile: StitchNet tended toward higher precision but lower recall. In a clinical setting, this means the model is less likely to produce false positives (flagging healthy tissue as diseased), though it may be less sensitive to very subtle or early-stage lesions.

Limitations and Future Directions

While promising, the current iteration of StitchNet has limitations. The authors noted an "identical style" problem where the model occasionally predicts the same style and mask for different unlabeled images, suggesting a need for better feature extraction. Additionally, the generative components currently use an older VGG16-style backbone; upgrading to more modern architectures like MobileNetV2 or Transformers could further bridge the performance gap with supervised methods. Future work will focus on balancing the reconstruction and segmentation losses to ensure the model captures the unique nuances of every individual patient scan.

Summary updated

Abstract

A recurring problem in image segmentation is a lack of labelled data. This problem is especially acute in the segmentation of lung computed tomography (CT) of patients with Coronavirus Disease 2019 (COVID-19). The reason for this is simple: the disease has not been prevalent long enough to generate a great number of labels. Semi-supervised learning promises a way to learn from data that is unlabelled and has seen tremendous advancements in recent years. However, due to the complexity of its label space, those advancements cannot be applied to image segmentation. That being said, it is this same complexity that makes it extremely expensive to obtain pixel-level labels, making semi-supervised learning all the more appealing. This study seeks to bridge this gap by proposing a novel model that utilizes the image segmentation abilities of deep convolution networks and the semi-supervised learning abilities of generative models for chest CT images of patients with the COVID-19. We propose a novel generative model called the shared variational autoencoder (SVAE). The SVAE utilizes a five-layer deep hierarchy of latent variables and deep convolutional mappings between them, resulting in a generative model that is well suited for lung CT images. Then, we add a novel component to the final layer of the SVAE which forces the model to reconstruct the input image using a segmentation that must match the ground truth segmentation whenever it is present. We name this final model StitchNet. We compare StitchNet to other image segmentation models on a high-quality dataset of CT images from COVID-19 patients. We show that our model has comparable performance to the other segmentation models. We also explore the potential limitations and advantages in our proposed algorithm and propose some potential future research directions for this challenging issue.
BibTeX
@article{zammit2022semi,
  author = {Judah Zammit and Daryl L. X. Fung and Qian Liu and Carson Kai-Sang Leung and Pingzhao Hu},
  title = {{Semi-supervised COVID-19 CT image segmentation using deep generative models}},
  journal = {BMC Bioinformatics},
  year = {2022},
  doi = {10.1186/s12859-022-04878-6},
}
Figure