TL;DR
This paper introduces CiT-Net, a novel hybrid architecture that combines Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) for medical image segmentation. By introducing a dynamic deformable convolution and a shifted-window adaptive complementary attention module, CiT-Net effectively captures both local and global image features, leading to improved segmentation accuracy with fewer parameters and lower computational costs compared to existing methods.
What Problem Does This Paper Address?
Medical image segmentation is crucial for diagnosis, treatment planning, and organ reconstruction. However, accurately segmenting complex medical images, which often suffer from noise, low contrast, and blurred edges, remains a significant challenge. Existing approaches often fall into two categories: pure CNNs, which excel at local feature extraction but struggle with long-range dependencies, and pure Transformers, which are good at global context but can overlook fine local details and require substantial data for training. Hybrid architectures aim to leverage the strengths of both, but previous attempts have faced challenges in adaptively learning local features and effectively modeling cross-dimensional relationships in medical images. This paper addresses these limitations by proposing a more robust and efficient hybrid architecture.
Methods
CiT-Net employs a dual-branch architecture, with one branch dedicated to CNNs and the other to Transformers, interacting through a cross-dimensional fusion mechanism.
- Dynamically Adaptive CNNs Branch: This branch incorporates a novel Dynamic Deformable Convolution (DDConv). Unlike standard convolutions with fixed kernels, DDConv can adaptively learn both the deformation offset and the weight coefficients of its kernels during training. This allows it to better capture irregular shapes and blurred edges in medical images, overcoming the limitations of fixed receptive fields and shared kernel parameters.
- Cross-dimensional Fusion Transformer Branch: This branch utilizes a Shifted Window Adaptive Complementary Attention Module ((S)W-ACAM). This module addresses the limitations of standard self-attention by not only considering spatial dependencies but also cross-dimensional relationships between spatial and channel information. It achieves this through four parallel attention branches. To manage computational complexity, it employs a shifted window approach and a compact convolutional projection to reduce spatial resolution and channel dimensions before attention calculation.
- Lightweight Perceptron Module (LPM): Inspired by the GhostNet architecture, the authors propose LPM to replace the standard MLP layers in Transformer blocks. This significantly reduces the number of parameters and computational cost of the Transformer branch, making the model more efficient and less prone to overfitting, even without extensive pre-training.
- Cross-dimensional Fusion: The CNN and Transformer branches are designed to interact, with local features from the CNNs fed to the Transformer decoder and global features from the Transformer fed to the CNN decoder, enabling a more comprehensive feature representation.
Datasets
The proposed CiT-Net was evaluated on two publicly available datasets:
- ISIC2018: A skin lesion segmentation dataset. The authors performed a five-fold cross-validation on the training set as the ground truth for the testing set was not released.
- LiTS (Liver Tumor Segmentation Challenge): A dataset for 3D CT liver scans. 100 scans were used for training and 31 for testing.
All images were resized to 224x224 for efficiency.
Key Insights and Ideas
- Adaptive Kernels for Local Features: The DDConv's ability to dynamically adjust kernel shape and weights is a key innovation for handling the inherent variability and noise in medical images, leading to better local feature extraction.
- Beyond Spatial Attention: The (S)W-ACAM's focus on cross-dimensional (spatial-channel) attention is crucial for medical images where subtle differences between target and background can be critical, and standard attention might miss these nuances.
- Efficiency Through Lightweight Design: The integration of LPM and the shifted-window approach in the Transformer branch demonstrates a strong focus on computational efficiency without sacrificing performance, making the model more practical for real-world applications.
- Synergy of CNNs and Transformers: The paper highlights how a carefully designed interaction between CNNs and Transformers can yield superior results by combining their complementary strengths, rather than just stacking them.
Key Results
CiT-Net demonstrated competitive and often superior performance on both ISIC2018 and LiTS datasets compared to state-of-the-art methods.
- ISIC2018: CiT-Net-T achieved the second-best segmentation effect with only 11.58M parameters and 4.53 GFLOPs, while CiT-Net-B also showed excellent results.
- LiTS: CiT-Net-B achieved the best segmentation performance, outperforming pre-trained Swin-Unet, TransUNet, and CvT by significant margins in Dice value, while using fewer parameters and computational costs.
- Efficiency: Both CiT-Net-T and CiT-Net-B achieved good results without pre-training, requiring fewer parameters and less computational cost than many comparable methods.
Limitations and Future Work
- Computational Cost of Hybrid Architecture: While CiT-Net is more efficient than many Transformer-based methods, the dual-branch architecture inherently has higher computational demands than a single-branch network. The authors mention this as a challenge that their LPM and (S)W-ACAM help to mitigate.
- Generalizability: While tested on skin lesions and liver tumors, further validation on a wider range of medical imaging modalities and anatomical structures would be beneficial.
- Future Work: The paper implicitly suggests that further exploration of adaptive attention mechanisms and efficient fusion strategies could lead to even more powerful segmentation models. The authors also highlight the potential of their architecture to perform well without extensive pre-training, which is a significant advantage for medical imaging where labeled data can be scarce.
Summary updated