CovSegNet: A Multi Encoder-Decoder Architecture for Improved Lesion Segmentation of COVID-19 Chest CT Scans

Tanvir Mahmud, Md Awsafur Rahman, Shaikh Anowarul Fattah, Sun-Yuan Kung

TL;DR

CovSegNet is a novel neural network architecture designed to precisely segment COVID-19 lesions in chest CT scans by combining a multi-stage "horizontal" encoder-decoder structure with a hybrid 2D-3D training strategy. It achieves state-of-the-art performance (up to a 94.1% Dice score) while significantly reducing computational complexity compared to traditional 3D segmentation models.

What Problem Does This Paper Address?

Accurate segmentation of lung lesions is critical for diagnosing COVID-19 severity, but manual inspection of 3D CT volumes is labor-intensive and prone to error. While deep learning—specifically the U-Net architecture—has become the standard for medical imaging, it faces three major hurdles in this context:

  1. Contextual Loss: Traditional pooling and upsampling operations discard fine-grained spatial details.
  2. Semantic Gaps: There is often a significant "gap" between the low-level features in the encoder and high-level features in the decoder, making convergence difficult.
  3. The 3D Trade-off: Processing entire 3D volumes is computationally expensive and often leads to vanishing gradient problems, while processing 2D slices individually loses vital inter-slice (depth) information.

Methods

The authors introduce CovSegNet, which moves beyond the "vertical" depth of standard U-Nets by adding "horizontal" expansion through multiple sequential encoder-decoder stages. Key innovations include:

  • Multi-Scale Fusion (MSF) & Pyramid Fusion (PF): Instead of simple skip connections, CovSegNet uses MSF modules to bridge the semantic gap between stages. These modules aggregate features from all preceding encoder/decoder levels and pass them through a Pyramid Fusion scheme that uses diverse windows of up-sampling and down-sampling to capture multi-scale context.
  • Advanced Transition Units: Rather than standard max-pooling, the "Down Transition" and "Up Transition" units incorporate feature maps from all previous levels of the current stage, ensuring that spatial information lost during scaling is recovered.
  • Two-Phase Hybrid Training:
    • Phase 1: A deep 2D network (CovSegNet2D) is trained on CT slices to generate ROI-enhanced (Region of Interest) volumes.
  • Phase 2: A shallower 3D network (CovSegNet3D) is then trained on these enhanced volumes. This "hybrid" approach captures 3D contextual information without the massive memory overhead of a full-scale 3D-only architecture.
  • Focal Tversky Loss: To handle the extreme class imbalance (lesions are often tiny compared to the whole lung), the authors utilize a Focal Tversky loss function, which penalizes false negatives more heavily and focuses on "hard" training examples.

Datasets

The model was evaluated on three datasets:

  1. Dataset-1: 20 CT volumes (1800+ slices) with expert annotations for lung and infection regions.
  2. Dataset-2: 110 axial CT images from 40 patients, featuring multi-class annotations (Ground Glass Opacity vs. Consolidation).
  3. Dataset-3 (Semantic Drone Dataset): To demonstrate the architecture's versatility, the authors tested it on a non-medical task involving urban scene segmentation from drone imagery.

Key Insights and Ideas

  • Horizontal > Vertical: The authors found that adding sequential encoder-decoder stages (horizontal expansion) provided better performance gains than simply making a single U-Net deeper. This allows the model to refine its predictions across multiple "passes."
  • Transition Memory: By making transition units "aware" of all previous levels, the network creates a form of short-term memory for spatial resolution, which is particularly helpful for detecting blurred or diffused lesion edges.
  • Efficiency via ROI: The hybrid 2D-3D approach proves that you don't need a massive 3D model to get 3D results. By using a 2D model to "crop" the problem down to the ROI first, the 3D component can be much lighter and more efficient.

Key Results

  • Superior Accuracy: CovSegNet outperformed several state-of-the-art models (U-Net, U-Net++, V-Net, Attention U-Net) across all metrics. It achieved a 94.1% Dice score on Dataset-1.
  • Significant Improvement: The model showed an average improvement of 8.4% in Dice measurement over traditional approaches.
  • Computational Efficiency: One variant, CovSegNet2D-v2, achieved better accuracy than the standard U-Net while using 94.8% fewer parameters. The hybrid model used only 7.8M parameters compared to Unet3D’s 90.3M, while delivering higher accuracy.
  • Generalizability: The model also achieved top-tier results on the non-medical Drone dataset, suggesting the architecture is robust for general semantic segmentation.

Limitations and Future Work

The authors acknowledge that while the results are promising, the study was limited by the relatively small size of available COVID-19 datasets at the time of writing. They suggest that future work should involve:

  • Larger, more diverse cohorts: Including data that accounts for different ages, health conditions, and geographical locations.
  • Generative Augmentation: Using GANs (Generative Adversarial Networks) to create synthetic data to further improve model robustness.
  • Severity Prediction: Extending the framework from simple segmentation to automated severity scoring and clinical outcome prediction.

Summary updated

Abstract

Automatic lung lesion segmentation of chest computer tomography (CT) scans is considered a pivotal stage toward accurate diagnosis and severity measurement of COVID-19. Traditional U-shaped encoder–decoder architecture and its variants suffer from diminutions of contextual information in pooling/upsampling operations with increased semantic gaps among encoded and decoded feature maps as well as instigate vanishing gradient problems for its sequential gradient propagation that result in suboptimal performance. Moreover, operating with 3-D CT volume poses further limitations due to the exponential increase of computational complexity making the optimization difficult. In this article, an automated COVID-19 lesion segmentation scheme is proposed utilizing a highly efficient neural network architecture, namely CovSegNet, to overcome these limitations. Additionally, a two-phase training scheme is introduced where a deeper 2-D network is employed for generating region-of-interest (ROI)-enhanced CT volume followed by a shallower 3-D network for further enhancement with more contextual information without increasing computational burden. Along with the traditional vertical expansion of Unet, we have introduced horizontal expansion with multistage encoder–decoder modules for achieving optimum performance. Additionally, multiscale feature maps are integrated into the scale transition process to overcome the loss of contextual information. Moreover, a multiscale fusion module is introduced with a pyramid fusion scheme to reduce the semantic gaps between subsequent encoder/decoder modules while facilitating the parallel optimization for efficient gradient propagation. Outstanding performances have been achieved in three publicly available datasets that largely outperform other state-of-the-art approaches. The proposed scheme can be easily extended for achieving optimum segmentation performances in a wide variety of applications. <p xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"><i><b>Impact Statement</b></i>—With lower sensitivity (60–70%), elongated testing time, and a dire shortage of testing kits, traditional RTPCR based COVID-19 diagnostic scheme heavily relies on postCT based manual inspection for further investigation. Hence, automating the process of infected lesions extraction from chestCT volumes will be major progress for faster accurate diagnosis of COVID-19. However, in challenging conditions with diffused, blurred, and varying shaped edges of COVID-19 lesions, conventional approaches fail to provide precise segmentation of lesions that can be deleterious for false estimation and loss of information. The proposed scheme incorporating an efficient neural network architecture (CovSegNet) overcomes the limitations of traditional approaches that provide significant improvement of performance (8.4% in averaged dice measurement scale) over two datasets. Therefore, this scheme can be an effective, economical tool for the physicians for faster infection analysis to greatly reduce the spread and massive death toll of this deadly virus through mass-screening.</p>
BibTeX
@article{mahmud2021covsegnet,
  author = {Tanvir Mahmud and Md Awsafur Rahman and Shaikh Anowarul Fattah and Sun-Yuan Kung},
  title = {{CovSegNet: A Multi Encoder-Decoder Architecture for Improved Lesion Segmentation of COVID-19 Chest CT Scans}},
  journal = {IEEE Trans. Artif. Intell.},
  year = {2021},
  url = {https://openreview.net/forum?id=9qLmVmLcAg},
}

Loading related content...

Figure