CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation

Tao Lei, Rui Sun, Xuan Wang, Yingbo Wang, Xi He, Asoke K. Nandi

Abstract

Medical image segmentation is a critical yet challenging task due to the inherent noise and irregular biological structures in clinical scans. While Convolutional Neural Networks (CNNs) excel at local feature extraction and Vision Transformers (ViTs) capture long-range dependencies, both have limitations in adaptability and computational efficiency. This paper introduces CiT-Net, a hybrid architecture that synergizes CNNs and ViTs. By leveraging Dynamic Deformable Convolutions (DDConv) and Cross-Dimensional Attention, CiT-Net effectively models irregular shapes and spatial-channel correlations. Experimental results on ISIC2018 and LiTS datasets demonstrate that CiT-Net achieves state-of-the-art performance with significantly fewer parameters and computational costs, all without the need for large-scale pre-training.

Introduction

Accurate segmentation of organs and lesions is fundamental to medical diagnosis and treatment planning. However, biological structures often exhibit high variability in shape and size, compounded by low contrast and noise in imaging modalities like CT and dermoscopy. Traditional CNN-based models, such as U-Net, rely on fixed-grid convolutions that lack the flexibility to adapt to irregular anatomical boundaries. Conversely, Vision Transformers offer powerful global context modeling but often overlook the intricate relationships between spatial dimensions and feature channels. Furthermore, Transformers are typically data-hungry and computationally intensive.

CiT-Net addresses these challenges by proposing a "hand-in-hand" dual-branch architecture. This design allows the CNN branch to provide local inductive biases while the Transformer branch captures global dependencies, creating a complementary system that is both robust and efficient.

Methods

The CiT-Net architecture consists of two parallel interactive branches: a Dynamically Adaptive CNN branch and a Cross-dimensional Fusion Transformer branch.

Architecture of CiT-Net showing the dual-branch interaction between CNN and Transformer components.

Key Innovations

  • Dynamic Deformable Convolution (DDConv): To overcome the rigidity of standard convolutions, DDConv learns sampling offsets and weight coefficients. This enables the network to dynamically adjust its receptive field to match the specific morphology of the target structure.
  • Shifted-Window Adaptive Complementary Attention Module ((S)W-ACAM): This module extends standard self-attention by incorporating branches that model correlations between spatial dimensions (height, width) and feature channels. This multi-dimensional approach improves the model's ability to distinguish targets from complex backgrounds.
  • Lightweight Perceptron Module (LPM): Inspired by efficient architectures like GhostNet, the LPM replaces standard heavy MLP layers. This significantly reduces the model's parameter count and computational overhead without sacrificing performance.
  • Interactive Feature Fusion: The two branches are not isolated; they exchange information throughout the encoding and decoding stages. This ensures that local geometric details from the CNN branch and global contextual information from the Transformer branch are effectively integrated.

Experiments and Results

CiT-Net was evaluated on the ISIC2018 skin lesion dataset and the LiTS liver tumor dataset. The model was trained from scratch, demonstrating its ability to learn effectively without ImageNet pre-training.

Performance and Efficiency

CiT-Net outperformed several state-of-the-art models, including TransUNet and Swin-Unet. On the ISIC2018 benchmark, the base version (CiT-Net-B) achieved a Dice Score (DI) of 91.23%. On the LiTS dataset, it reached a DI of 96.82%.

Performance comparison of CiT-Net against state-of-the-art CNN and Transformer models on the ISIC2018 dataset.

Beyond accuracy, CiT-Net demonstrates exceptional efficiency. The tiny version (CiT-Net-T) utilizes only 11.58 million parameters and 4.53 GFLOPs—roughly one-fourth the parameters of Swin-Unet and a fraction of TransUNet's computational cost. This efficiency makes it a strong candidate for clinical deployment on standard hardware.

Conclusion

CiT-Net successfully bridges the gap between CNNs and Vision Transformers for medical image segmentation. By introducing dynamic adaptability and cross-dimensional attention, the model achieves superior accuracy and efficiency. Future research will explore extending this architecture to native 3D data processing and validating its performance across a broader range of medical imaging modalities.

Summary updated

BibTeX
@inproceedings{lei2023cit,
  author = {Tao Lei and Rui Sun and Xuan Wang and Yingbo Wang and Xi He and Asoke K. Nandi},
  title = {{CiT-Net: Convolutional Neural Networks Hand in Hand with Vision Transformers for Medical Image Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.24963/ijcai.2023/113},
}
Figure