TransDeepLab: Convolution-Free Transformer-Based DeepLab v3+ for Medical Image Segmentation

Reza Azad, Moein Heidari, Moein Shariatnia, Ehsan Khodapanah Aghdam, Sanaz Karimijafarbigloo, Ehsan Adeli, Dorit Merhof

Abstract

Medical image segmentation requires a precise balance between capturing fine-grained local details and understanding broad global context. While Convolutional Neural Networks (CNNs) have long been the standard for this task, their inherent locality bias often limits their ability to model long-range spatial dependencies. TransDeepLab introduces the first "pure" Transformer-based adaptation of the iconic DeepLabv3+ architecture. By replacing traditional convolutions with hierarchical Swin-Transformer blocks, the model achieves state-of-the-art performance on organ and skin lesion segmentation while maintaining a significantly smaller parameter footprint than competing hybrid models.

Introduction

Precise segmentation—the task of outlining organs or lesions at the pixel level—is critical for computer-aided diagnosis and treatment planning. Traditionally, architectures like U-Net and DeepLabv3+ have relied on CNNs to extract features. However, the fixed receptive field of convolution kernels makes it difficult for these models to capture the global relationships necessary for accurate segmentation in complex anatomical environments, often leading to errors in boundary definition for difficult-to-segment organs.

Vision Transformers (ViTs) were introduced to overcome these limitations by utilizing self-attention to model global dependencies. However, standard ViTs face challenges regarding quadratic computational complexity and a relative weakness in capturing low-level pixel information. Existing "hybrid" models attempt to bridge this gap by combining CNN and Transformer layers, but they often result in massive, parameter-heavy architectures. TransDeepLab addresses these issues by reimagining the DeepLabv3+ framework as a purely Transformer-based system, leveraging the efficiency and hierarchical nature of Swin-Transformers.

Methodology

TransDeepLab maintains the structural logic of the original DeepLabv3+ but replaces every convolutional component with a Transformer-based equivalent, creating a convolution-free pipeline.

Swin-Transformer Backbone

The model utilizes a Swin-Transformer backbone to extract hierarchical features. Unlike standard ViTs, Swin-Transformers employ "shifted windows" to compute self-attention. This mechanism provides the global modeling advantages of Transformers while maintaining linear computational complexity relative to image size, making it feasible for high-resolution medical imaging.

Swin Spatial Pyramid Pooling (SSPP)

A core feature of the original DeepLabv3+ is Atrous Spatial Pyramid Pooling (ASPP), which uses dilated convolutions to capture multi-scale information. TransDeepLab introduces the Swin Spatial Pyramid Pooling (SSPP) module. By varying the window sizes within the Swin blocks, the SSPP module captures local textures with smaller windows and global shapes with larger windows. This effectively translates the concept of "dilation rates" into the Transformer domain, allowing for multi-scale reasoning without convolutional filters.

Cross-Contextual Attention and Decoder

To integrate features from different SSPP levels, the authors developed a Cross-Contextual Attention module. This module applies both channel and spatial attention to adaptively weight features, ensuring that the most relevant information is prioritized during fusion. The decoder then utilizes "patch expanding" layers to upsample features back to the original resolution, ensuring the entire pipeline remains convolution-free.

Experimental Results

The performance of TransDeepLab was evaluated on the Synapse Multi-Organ Segmentation dataset (abdominal CT scans) and several skin lesion datasets (ISIC 2017, 2018, and PH2).

Multi-Organ Segmentation

On the Synapse dataset, TransDeepLab achieved a Dice Similarity Coefficient (DSC) of 80.16%. Notably, it significantly outperformed the CNN-based DeepLabv3+ in terms of Hausdorff Distance (21.25 vs. 39.95), indicating much higher boundary precision and fewer outliers in the segmentation masks. Qualitative results suggest that the improved global context allows the model to produce cleaner boundaries for challenging organs like the pancreas.

Skin Lesion Segmentation

The model set new benchmarks for skin lesion segmentation, achieving a DSC of 0.9239 on the ISIC 2017 dataset. This demonstrates the model's versatility across different medical imaging modalities, from CT scans to dermoscopy images.

Efficiency and Complexity

A standout metric for TransDeepLab is its efficiency. The model contains only 21.14 million parameters. In comparison, the popular hybrid model TransUNet requires 105 million parameters. Achieving superior accuracy with approximately one-fifth of the parameters represents a significant advancement for practical clinical applications where deployment on standard hardware is often required.

Discussion and Conclusion

TransDeepLab demonstrates that a "pure" Transformer architecture can outperform both traditional CNNs and complex hybrid models in medical image segmentation. By replacing dilated convolutions with multi-scale windowed attention, the model captures the necessary global context without sacrificing the local detail required for precise medical boundaries.

While the current 2D implementation shows remarkable results, the authors note that the effectiveness of the SSPP module is tied to input resolution. Future work may involve extending this architecture to 3D Swin-Transformers to capture volumetric context directly from 3D medical data. Additionally, further refinements to the cross-contextual attention mechanism could continue to drive down parameter counts while maintaining high segmentation accuracy.

Summary updated

BibTeX
@inproceedings{azad2022transdeeplab,
  author = {Reza Azad and Moein Heidari and Moein Shariatnia and Ehsan Khodapanah Aghdam and Sanaz Karimijafarbigloo and Ehsan Adeli and Dorit Merhof},
  title = {{TransDeepLab: Convolution-Free Transformer-Based DeepLab v3+ for Medical Image Segmentation}},
  booktitle = {PRIME @ MICCAI},
  year = {2022},
  doi = {10.1007/978-3-031-16919-9\_9},
}
Figure