Spatio-Temporal Hybrid Fusion of CAE and SWin Transformers for Lung Cancer Malignancy Prediction

Sadaf Khademi, Shahin Heidarian, Parnian Afshar, Farnoosh Naderkhani, Anastasia Oikonomou, Konstantinos N. Plataniotis, Arash Mohammadi

TL;DR

This paper introduces CAET-SWin, a novel hybrid deep learning framework that fuses temporal and spatial features from CT scans to predict lung adenocarcinoma malignancy. By leveraging Convolutional Auto-Encoders (CAE) for inter-slice temporal information and Shifted Window (SWin) Transformers for intra-slice spatial details, CAET-SWin achieves improved accuracy and reliability in malignancy prediction with minimal expert involvement.

What Problem Does This Paper Address?

Lung cancer remains a leading cause of mortality, with Lung Adenocarcinoma (LUAC) being the most prevalent subtype. Accurately classifying LUACs as pre-invasive, minimally invasive, or invasive is crucial for timely and appropriate treatment planning, potentially reducing unnecessary surgeries. While chest CT scans are the primary imaging tool, radiologists' manual analysis is subjective, time-consuming, and prone to errors. Existing automated methods often rely on high-resolution, thin-slice CT scans, which are not always available, or require extensive manual annotation. This paper addresses the need for an automated, accurate, and robust framework that can predict LUAC malignancy from standard, non-thin CT scans, minimizing the need for expert input.

Methods

The core of the CAET-SWin framework is its hybrid architecture, which combines two parallel self-attention-based paths to extract distinct types of features:

  • CAE-Transformer Path (Temporal Features): This path uses a Convolutional Auto-Encoder (CAE) to generate compressed, informative feature maps from CT slices. Crucially, instead of the standard patch-based approach in Vision Transformers (ViT), the CAE's output is directly fed into a modified Transformer encoder. This encoder, utilizing Multi-head Self-Attention (MSA), captures temporal dependencies between CT slices, effectively modeling the sequence of images.
  • SWin-Transformer Path (Spatial Features): This path employs a hierarchical Vision Transformer, specifically the Shifted Window (SWin) Transformer. It focuses on extracting spatial features within each CT slice by applying self-attention within local, shifted windows. This approach is designed to efficiently capture local nodule characteristics.

These two paths operate in parallel, extracting complementary temporal and spatial information. Finally, a Fusion Path concatenates the feature vectors from both paths and passes them through a series of fully connected layers to produce the final malignancy prediction. A key advantage highlighted is that this framework does not require detailed nodule segmentation, only an approximate region of interest.

Datasets

The study utilizes an in-house dataset consisting of 114 pathologically proven Sub-Solid Nodules (SSNs) from non-thin volumetric CT scans. These nodules were segmented and reviewed by two experienced thoracic radiologists. The SSNs were classified into three histological categories: pre-invasive lesions (atypical adenomatous hyperplasia and adenocarcinoma in situ), minimally invasive, and invasive pulmonary adenocarcinomas. For the purpose of this study, the pre-invasive and minimally invasive categories were grouped together as "pre-invasive," while the invasive nodules formed the "invasive" class. The paper also mentions using the public LIDC-IDRI dataset for pre-training the CAE model.

Key Insights and Ideas

  • Synergistic Fusion of Temporal and Spatial Features: The paper demonstrates that combining inter-slice (temporal) and intra-slice (spatial) information through parallel deep learning paths significantly enhances malignancy prediction accuracy compared to using either feature type alone.
  • Modified Transformer for Temporal Data: Adapting the Transformer architecture by replacing patch embedding with CAE features for temporal modeling is an effective way to leverage sequential CT data without relying on traditional recurrent networks.
  • Efficiency of SWin Transformer for Spatial Features: The SWin Transformer's hierarchical structure and shifted window mechanism prove adept at capturing local spatial patterns within CT slices, contributing to robust feature extraction.
  • Reduced Expert Annotation Burden: The framework's ability to perform well with approximate nodule regions, rather than precise segmentations, is a significant practical advantage, reducing the time and effort required from radiologists.

Key Results

The CAET-SWin framework achieved promising results on the in-house dataset:

  • Accuracy: 82.65%
  • Sensitivity: 83.66%
  • Specificity: 81.66%

These results were obtained using a 10-fold cross-validation. The paper highlights that CAET-SWin significantly outperformed its constituent parts (CAET and SWin paths individually) and a radiomics-based model from a previous study [8]. Specifically, the sensitivity of 83.66% was approximately 4% higher than the reference study, indicating a better ability to correctly identify malignant nodules. The confidence intervals for CAET-SWin were also narrower, suggesting greater reliability.

Limitations and Future Work

  • Dataset Size: While the in-house dataset of 114 SSNs is pathologically proven, it is relatively small. The authors suggest that investigating the effects of embedding radiomics and morphological features into the CAET-SWin framework is a fruitful direction for future research.
  • Generalizability: The performance on an in-house dataset might not directly translate to other datasets or clinical settings without further validation.
  • Future Directions: The authors explicitly mention exploring the integration of radiomics and morphological features as a potential avenue for further improvement.

Summary updated

Abstract

The paper proposes a novel hybrid discovery Radiomics framework that simultaneously integrates temporal and spatial features extracted from non-thin chest Computed Tomography (CT) slices to predict Lung Adenocarcinoma (LUAC) malignancy with minimum expert involvement. Lung cancer is the leading cause of mortality from cancer worldwide and has various histologic types, among which LUAC has recently been the most prevalent. LUACs are classified as pre-invasive, minimally invasive, and invasive adenocarcinomas. Timely and accurate knowledge of the lung nodules malignancy leads to a proper treatment plan and reduces the risk of unnecessary or late surgeries. Currently, chest CT scan is the primary imaging modality to assess and predict the invasiveness of LUACs. However, the radiologists’ analysis based on CT images is subjective and suffers from a low accuracy compared to the ground truth pathological reviews provided after surgical resections. The proposed hybrid framework, referred to as the CAET-SWin, consists of two parallel paths: (i) The Convolutional Auto-Encoder (CAE) Transformer path that extracts and captures informative features related to inter-slice relations via a modified Transformer architecture, and; (ii) The Shifted Window (SWin) Transformer path, which is a hierarchical vision transformer that extracts nodules’ related spatial features from a volumetric CT scan. Extracted temporal (from the CAET path) and spatial (from the SWin path) are then fused through a fusion path to classify LUACs. Experimental results on our in-house dataset of 114 pathologically proven SubSolid Nodules (SSNs) demonstrate that the CAET-SWin significantly improves reliability of the invasiveness prediction task while achieving an accuracy of 82.65%, sensitivity of 83.66%, and specificity of 81.66% using 10-fold cross-validation.
BibTeX
@inproceedings{khademi2023spatio,
  author = {Sadaf Khademi and Shahin Heidarian and Parnian Afshar and Farnoosh Naderkhani and Anastasia Oikonomou and Konstantinos N. Plataniotis and Arash Mohammadi},
  title = {{Spatio-Temporal Hybrid Fusion of CAE and SWin Transformers for Lung Cancer Malignancy Prediction}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=Uk-zUJrtAc},
}
Figure