DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation

Feilong Tang, Qiming Huang, Jinfeng Wang, Xianxu Hou, Jionglong Su, Jingxin Liu

Abstract

Transformer-based models have been widely demonstrated to be successful in computer vision tasks by modelling long-range dependencies and capturing global representations. However, they are often dominated by features of large patterns leading to the loss of local details (e.g., boundaries and small objects), which are critical in medical image segmentation. To alleviate this problem, we propose a Dual-Aggregation Transformer Network called DuAT, which is characterized by two innovative designs, namely, the Global-to-Local Spatial Aggregation (GLSA) and Selective Boundary Aggregation (SBA) modules. The GLSA has the ability to aggregate and represent both global and local spatial features, which are beneficial for locating large and small objects, respectively. The SBA module is used to aggregate the boundary characteristic from low-level features and semantic information from high-level features for better preserving boundary details and locating the re-calibration objects. Extensive experiments in six benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods in the segmentation of skin lesion images, and polyps in colonoscopy images. In addition, our approach is more robust than existing methods in various challenging situations such as small object segmentation and ambiguous object boundaries.

Introduction

Medical image segmentation—the process of digitally isolating tumors, polyps, or organs from clinical scans—is a high-stakes task where precision is paramount. While Transformer-based models have recently revolutionized computer vision due to their ability to model long-range dependencies and understand global context, they often suffer from a form of "farsightedness." Their inherent focus on global patterns can lead to the neglect of fine-grained local details, such as precise tissue boundaries and very small objects. In clinical settings, missing a tiny early-stage polyp or misidentifying the edge of a lesion can have significant diagnostic consequences.

The DuAT (Dual-Aggregation Transformer) architecture addresses these limitations by introducing a "macro-to-micro" upgrade to the standard Transformer framework. By balancing global representation with local detail preservation, DuAT aims to provide a more robust solution for challenging medical imaging scenarios.

Methods

The DuAT architecture is built upon a Pyramid Vision Transformer (PVT) backbone, which serves to extract multi-scale features from the input image. The core innovation of the network lies in its two specialized "Dual-Aggregation" modules designed to bridge the gap between global and local information.

The DuAT architecture featuring Global-to-Local Spatial Aggregation (GLSA) and Selective Boundary Aggregation (SBA) modules.

Global-to-Local Spatial Aggregation (GLSA)

To prevent the model from being dominated solely by large-scale patterns, the GLSA module splits feature channels into two distinct streams. One stream utilizes a Global Context (GC) block to capture long-range dependencies, while the parallel stream employs multiple depth-wise convolutions to focus on fine-grained local details. By fusing these streams, the model maintains a comprehensive understanding of the image without losing track of small, low-contrast objects.

Selective Boundary Aggregation (SBA)

The SBA module is specifically designed to address the problem of ambiguous boundaries. It utilizes a Re-calibration Attention Unit (RAU) to selectively aggregate low-level boundary details from early network layers with high-level semantic information from deeper layers. This process allows the model to "re-calibrate" its initial predictions, sharpening blurry edges and improving the accuracy of the final segmentation mask.

Experiments and Datasets

The researchers evaluated DuAT across six diverse benchmark datasets to test its learning capacity and generalization ability:

  • Polyp Segmentation: Kvasir-SEG, CVC-ClinicDB, CVC-ColonDB, ETIS-LaribPolypDB, and CVC-300 (EndoScene).
  • Skin Lesion Segmentation: ISIC-2018.
  • Nuclei Segmentation: 2018 Data Science Bowl (2018-DSB).

A key focus of the evaluation was the model's performance on "unseen" datasets, such as ETIS, which contains particularly small and difficult-to-detect polyps, ensuring the model's utility in real-world clinical environments where data variability is high.

Key Results

DuAT consistently outperformed state-of-the-art (SOTA) models across all tested metrics, including mean Dice (mDice), mean Intersection over Union (mIoU), and Mean Absolute Error (MAE).

Qualitative comparison of DuAT against state-of-the-art methods across various medical imaging datasets.
  • Superior Generalization: On the challenging ETIS dataset, DuAT achieved a mean Dice score of 82.2%, outperforming the previous leading model, Polyp-PVT, by a significant 3.5%.
  • Robustness to Scale: The model demonstrated high accuracy even when target objects occupied less than 5% of the total image area—a scenario where traditional models like U-Net often fail.
  • Efficiency: Despite its Transformer-based architecture, DuAT is computationally efficient. With approximately 24.9M parameters, it is significantly smaller than competitors like TransUNet (~93M parameters), making it more viable for practical deployment.
  • Visual Clarity: Qualitative analysis shows that DuAT produces cleaner, more accurate masks for "camouflaged" objects that blend into the background, avoiding the fragmented or noisy outputs common in earlier architectures.

Conclusion and Future Work

By effectively balancing the "big picture" with "fine details," DuAT provides a more reliable tool for medical image segmentation. While currently optimized for 2D imaging (such as dermoscopy and colonoscopy), future work will focus on extending these dual-aggregation modules to 3D spatial-temporal data, such as MRI and CT volumes. The success of DuAT suggests that specialized attention mechanisms can overcome the traditional weaknesses of Transformers in medical AI, potentially leading to earlier and more accurate clinical diagnoses.

Summary updated

Abstract

Transformer-based models have been widely demonstrated to be successful in computer vision tasks by modelling long-range dependencies and capturing global representations. However, they are often dominated by features of large patterns leading to the loss of local details (e.g., boundaries and small objects), which are critical in medical image segmentation. To alleviate this problem, we propose a Dual-Aggregation Transformer Network called DuAT, which is characterized by two innovative designs, namely, the Global-to-Local Spatial Aggregation (GLSA) and Selective Boundary Aggregation (SBA) modules. The GLSA has the ability to aggregate and represent both global and local spatial features, which are beneficial for locating large and small objects, respectively. The SBA module is used to aggregate the boundary characteristic from low-level features and semantic information from high-level features for better preserving boundary details and locating the re-calibration objects. Extensive experiments in six benchmark datasets demonstrate that our proposed model outperforms state-of-the-art methods in the segmentation of skin lesion images, and polyps in colonoscopy images. In addition, our approach is more robust than existing methods in various challenging situations such as small object segmentation and ambiguous object boundaries.
BibTeX
@inproceedings{tang2023duat,
  author = {Feilong Tang and Qiming Huang and Jinfeng Wang and Xianxu Hou and Jionglong Su and Jingxin Liu},
  title = {{DuAT: Dual-Aggregation Transformer Network for Medical Image Segmentation}},
  booktitle = {PRCV},
  year = {2023},
  doi = {10.1007/978-981-99-8469-5\_27},
}
Figure