Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation

Wang Dong, Zixiang Wang, Ling Chen, Hongfeng Xiao, Bo YangSensors 2023 · DOI

TL;DR

This paper introduces a novel "Cross-Parallel Transformer" (C-PT) module to improve medical image segmentation. By redesigning the Vision Transformer (ViT) architecture from a sequential to a parallel structure, the authors achieve better accuracy and significantly reduced computational costs compared to existing methods, making advanced segmentation more accessible.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis and treatment planning. While Convolutional Neural Networks (CNNs) excel at capturing local features, they struggle with global context. Transformers, particularly Vision Transformers (ViTs), have emerged as powerful tools for capturing long-range dependencies, leading to improved segmentation accuracy. However, traditional ViT implementations in medical imaging often use sequential blocks, which can lead to increased model complexity, higher computational demands, and slower training. Existing parallel ViT approaches have also faced challenges with efficiency and maintaining semantic representation. This paper aims to bridge this gap by proposing a parallel Transformer architecture that enhances both accuracy and efficiency in medical image segmentation.

Methods

The core innovation is the Cross-Parallel Transformer (C-PT) module. This module refines the Vision Transformer by:

  • Parallel Architecture: Instead of a sequential arrangement, the C-PT module uses parallel branches for feature processing. This increases the "width" of the network, allowing for more efficient feature extraction.
  • Cross-Parallel Multi-Head Self-Attention (C-PMHSA): This novel attention mechanism enhances feature extraction by incorporating self-correlated feature attention and channel feature interaction between parallel branches. It aims to improve long-range modeling and high-order spatial interactions.
  • Streamlined Feed-Forward Network (FFN): The computationally expensive FFN in the standard Transformer is simplified. The authors replace it with a more efficient activation function block (Layer Normalization and GELU), significantly reducing computational demands and parameters.

These C-PT modules are integrated into a U-Net-like architecture, replacing the sequential ViT blocks of the baseline TransUNet model. This results in two proposed models: PTransUNet (PT), which uses a standard parallel ViT, and C-PTransUNet (C-PT), which incorporates the novel C-PT module.

Datasets

The authors evaluated their models on two publicly available datasets:

  • Synapse Dataset: This dataset consists of CT scans of eight abdominal organs from 30 cases, totaling 3779 cross-sectional slices. It's used for multi-organ segmentation tasks.
  • ACDC Dataset: This dataset comprises cardiovascular magnetic resonance images from 100 cases, with 1902 cross-sectional slices, focusing on segmenting the right ventricle (RV), left ventricle (LV), and myocardial layer (MYO).

Key Insights and Ideas

  • Parallelism is Key for Efficiency: Moving from sequential to parallel Transformer blocks is crucial for reducing computational burden without sacrificing accuracy. This is a significant departure from the common practice of simply increasing network depth.
  • Cross-Attention Enhances Feature Fusion: The C-PMHSA's ability to fuse information between parallel branches is vital for capturing richer, more robust feature representations. This "self-crossover attention" helps overcome limitations of purely parallel or sequential attention.
  • Simplifying FFNs is Effective: The FFN is often a bottleneck in Transformers. Streamlining it with just normalization and activation functions proves effective for medical image segmentation, drastically cutting down parameters and FLOPs.
  • Depth vs. Width Trade-off: The paper explores how depth and parallelism interact. While increasing depth can improve performance, it also risks feature saturation and overfitting. The C-PT module's parallel design offers a way to achieve better performance by increasing "width" rather than just depth.

Key Results

  • Improved Accuracy: The C-PT model achieved a significant improvement in average Dice Similarity Coefficient (DSC) accuracy of 3.25% compared to the baseline TransUNet on the Synapse dataset. The PT model also showed improvement (0.87%).
  • Reduced Computational Cost: The C-PT model demonstrated a remarkable 29% decrease in parameter count and a 21.4% reduction in FLOPs compared to the baseline, making it much more efficient. The PT model maintained similar parameter counts and FLOPs to the baseline.
  • Competitive Performance: Both PT and C-PT models outperformed several other mainstream segmentation models on the Synapse dataset, particularly in terms of DSC and HD95 metrics.
  • Robustness: On the ACDC dataset, the C-PT model showed superior segmentation performance and robustness, especially in segmenting the Myo region.
  • Ablation Studies: Experiments confirmed the effectiveness of the DD layer and the streamlined activation function block, showing that even a single DD layer can improve performance.

Limitations and Future Work

  • Parallelism and Stability: While increasing parallelism can improve segmentation accuracy, the authors note that it can also impact segmentation stability.
  • 3D Segmentation: The current work focuses on 2D medical image segmentation. A future direction is to explore the implementation of the C-PT module in 3D medical image segmentation.
  • Generalizability: The authors suggest applying the C-PT module to other network architectures to further explore its performance benefits.

Summary updated

Abstract

Medical image segmentation primarily utilizes a hybrid model consisting of a Convolutional Neural Network and sequential Transformers. The latter leverage multi-head self-attention mechanisms to achieve comprehensive global context modelling. However, despite their success in semantic segmentation, the feature extraction process is inefficient and demands more computational resources, which hinders the network's robustness. To address this issue, this study presents two innovative methods: PTransUNet (PT model) and C-PTransUNet (C-PT model). The C-PT module refines the Vision Transformer by substituting a sequential design with a parallel one. This boosts the feature extraction capabilities of Multi-Head Self-Attention via self-correlated feature attention and channel feature interaction, while also streamlining the Feed-Forward Network to lower computational demands. On the Synapse public dataset, the PT and C-PT models demonstrate improvements in DSC accuracy by 0.87% and 3.25%, respectively, in comparison with the baseline model. As for the parameter count and FLOPs, the PT model aligns with the baseline model. In contrast, the C-PT model shows a decrease in parameter count by 29% and FLOPs by 21.4% relative to the baseline model. The proposed segmentation models in this study exhibit benefits in both accuracy and efficiency.
BibTeX
@inproceedings{dong2023cross,
  author = {Wang Dong and Zixiang Wang and Ling Chen and Hongfeng Xiao and Bo Yang},
  title = {{Cross-Parallel Transformer: Parallel ViT for Medical Image Segmentation}},
  booktitle = {Sensors},
  year = {2023},
  doi = {10.3390/s23239488},
}
Figure