Multitask connected U-Net: automatic lung cancer segmentation from CT images using PET knowledge guidance

Lu Zhou, Chaoyong Wu, Yiheng Chen, Zhicheng ZhangFrontiers Artif. Intell 2024 · DOI

TL;DR

This paper introduces a novel "multitask connected U-Net" that leverages knowledge from PET scans to improve lung cancer segmentation on CT images, even without direct PET data. By employing a teacher-student framework, the model learns to generate "pseudo-PET" images from CT scans, effectively guiding the segmentation process and achieving state-of-the-art results on several benchmarks.

What Problem Does This Paper Address?

Lung cancer remains a leading cause of cancer-related mortality worldwide. Accurate segmentation of lung tumors on medical images is crucial for diagnosis, staging, and treatment planning. While CT scans offer high spatial resolution and are cost-effective, they often struggle with differentiating malignant from non-malignant lesions due to low contrast. PET/CT imaging provides complementary functional information, significantly improving tumor characterization. However, the high cost and radiation exposure associated with PET scans limit their widespread use. This creates a need for methods that can achieve PET-level segmentation accuracy using only CT data, or by intelligently incorporating PET knowledge without requiring actual PET images for every patient.

Methods

The core of this work is the Multitask Connected U-Net architecture, built upon a teacher-student framework.

  • Multitask Connected U-Net: This architecture consists of two U-Net models. The first U-Net is trained to generate "pseudo-PET" images from CT scans. The second U-Net performs the actual tumor segmentation. Crucially, the second U-Net incorporates features from the first U-Net's upsampling path via a "skipUnetConnect" to leverage the generated PET information.
  • Teacher-Student Framework: This framework is employed for semi-supervised learning.
  • Student U-Net: Learns to generate pseudo-PET images (PET-SA) from CT images (Data A) using Mean Squared Error (MSE) loss. It also learns tumor segmentation (TSI) from CT images (Data B) using focal and dice losses.
  • Teacher U-Net: Learns to generate pseudo-PET images (PET-TB) from CT images (Data B) using rotated versions of the CT images.
  • Consistency Loss: The student's pseudo-PET (PET-SB) is compared to the teacher's pseudo-PET (PET-TB) using a consistency loss to ensure stable and reliable pseudo-PET generation. The overall loss function combines MSE loss for pseudo-PET generation, focal and dice losses for segmentation, and consistency loss.
  • Tumor Area Detection: To further refine segmentation, a tumor area detection module is introduced. This involves rotating CT slices, clustering-based detection to identify potential tumor regions, and filtering non-critical areas to focus the segmentation on relevant regions.

Datasets

The study utilized four publicly available datasets:

  • NSCLC+Radiomics (NSCLC+Rad): Contains CT and PET images with corresponding CT segmentation labels. Used for model training.
  • NSCLC-Radiomics (NSCLC-Rad): Contains CT images with lung cancer segmentation labels. Used for external validation.
  • NSCLC-Radiomics-Interobserver1 (NSCLC-Rad-Int): Contains CT images and segmentation labels from multiple experts. Used for external validation.
  • Medical Segmentation Decathlon Task06 (MSD Task06): A well-known NSCLC segmentation competition dataset with CT images and segmentation labels. Used for external validation.

The NSCLC+Rad dataset was split into training (80%), validation (10%), and testing (10%) sets. The other three datasets were used for external validation. Images were standardized to 512x512 resolution and then cropped to 288x288 for training.

Key Insights and Ideas

  • Pseudo-PET Generation for Guidance: The core innovation is using a learned "pseudo-PET" signal derived solely from CT data to guide tumor segmentation. This bypasses the need for actual PET scans, addressing cost and accessibility issues.
  • Teacher-Student for Robustness: The teacher-student framework, particularly the consistency loss between teacher and student pseudo-PET, helps stabilize the pseudo-PET generation and ensures the student model learns robust features.
  • Focusing on Critical Regions: The tumor area detection method effectively filters out irrelevant areas, allowing the model to concentrate its segmentation efforts on the most promising regions, leading to improved accuracy.
  • Multimodal Fusion without Multimodal Input: The model demonstrates that knowledge from one modality (PET) can be effectively transferred and utilized for segmentation on another modality (CT) even when the target modality's input is the only one available during inference.

Key Results

The proposed multitask connected U-Net achieved state-of-the-art performance across multiple datasets:

  • Dice Coefficient: The model achieved an average Dice coefficient of 0.56, outperforming existing methods like Segformer (0.51), Transformer (0.50), and UctransNet (0.43) on average across tested datasets.
  • Specific Dataset Performance:
    • On the challenging MSD Task06 dataset, the model achieved a Dice score of 0.66 and an IOU of 0.55, with the lowest HD95% of 13.05.
  • On the NSCLC+Rad dataset, it achieved a Dice score of 0.64.
  • On NSCLC-Rad-Int, it achieved a Dice score of 0.57.
  • Ablation Studies: Integrating PET guidance and tumor area detection significantly improved performance compared to using CT images alone. The combined approach yielded the best results across most metrics and datasets.

Limitations and Future Work

  • Single Tumor Focus: The current tumor area detection method is primarily designed for single tumors. Future work should focus on validating its effectiveness for localizing multiple tumors.
  • Cost-Effectiveness and Safety: While the method aims to alleviate cost and radiation risks, further research is needed to evaluate its long-term cost-effectiveness and safety implications.
  • Generalizability to Multiple Tumors: The paper acknowledges that the tumor area detection is constrained to single tumors and warrants further verification for multi-tumor scenarios.
  • Broader Validation: The authors suggest future work will focus on further refining the model and validating its performance on larger and more diverse datasets.

Summary updated

BibTeX
@article{zhou2024multitask,
  author = {Lu Zhou and Chaoyong Wu and Yiheng Chen and Zhicheng Zhang},
  title = {{Multitask connected U-Net: automatic lung cancer segmentation from CT images using PET knowledge guidance}},
  journal = {Frontiers Artif. Intell.},
  year = {2024},
  doi = {10.3389/frai.2024.1423535},
}
Figure