Attention Mechanisms in Medical Image Segmentation: A Survey

Yutong Xie, Bing Yang, Qingbiao Guan, Jianpeng Zhang, Qi Wu, Yong XiaarXiv 2023 · DOI · arXiv

Introduction

Medical image segmentation—the precise labeling of organs, tissues, or lesions in scans such as CT, MRI, and ultrasound—is a cornerstone of modern computer-aided diagnosis. However, this task is fraught with challenges: low contrast between soft tissues, significant variability in pathology appearance, and a scarcity of expert-annotated data. While standard Convolutional Neural Networks (CNNs) like the U-Net have revolutionized the field, they often struggle with long-range dependencies and global context.

Attention mechanisms have emerged as a powerful solution, mimicking human visual cognition by adaptively weighting important regions while suppressing irrelevant background noise. This survey provides a comprehensive roadmap of the field, analyzing over 300 studies to categorize the rapid evolution of these techniques.

A Structured Taxonomy of Attention

The research landscape is broadly divided into two primary "families" based on their architectural foundations: Non-Transformer and Transformer-based attention. This structured approach helps researchers navigate the diverse strategies used to enhance segmentation performance.

A structured taxonomy of attention mechanisms in medical image segmentation, categorizing methods by architecture and application.

Non-Transformer Attention

These are typically modular "plug-ins" integrated into existing CNN frameworks. They include:

  • Channel Attention: Focuses on "what" features are most relevant by reweighting feature maps.
  • Spatial Attention: Focuses on "where" the target object is located, which is crucial for identifying small lesions.
  • Temporal Attention: Designed for video sequences or multi-slice volumetric data to capture consistency over time or depth.

Transformer-based Attention

Since 2020, the field has shifted toward self-attention as a core backbone. This includes Hybrid models that combine CNNs for local texture extraction with Transformers for global context, and Pure Transformers (e.g., Swin-Unet) that replace convolutional layers entirely to model long-range spatial relationships.

The Rapid Growth of the Field

The adoption of attention mechanisms in medical imaging has seen an exponential surge. As documented in the survey, the number of related publications grew from nearly zero in 2017 to a peak of over 220 papers in 2022 alone.

The number of papers published on attention-based medical image segmentation has grown dramatically since 2017, with a significant portion appearing in top-tier venues like MICCAI and Arxiv.

This growth reflects the transition from simple modular additions to sophisticated Transformer-based architectures that now dominate benchmarks like BraTS (brain tumors) and ACDC (cardiac segmentation).

Key Insights and Clinical Implications

Several critical findings emerge from the surveyed literature:

  • Spatial Dominance: Spatial attention is more prevalent than channel attention in medical tasks, likely because accurately defining boundaries and locating small pathologies is the primary bottleneck in clinical scans.
  • The Skip Connection "Sweet Spot": Integrating attention into the skip connections of a U-Net is particularly effective. It filters out noisy, low-level encoder features before they are fused with high-level decoder features, leading to cleaner segmentations.
  • The Hybrid Advantage: Despite the trend toward pure Transformers, hybrid CNN-Transformer models often yield the best results. They leverage the "inductive bias" of CNNs for local patterns while using Transformers to capture the global spatial relationships that CNNs inherently miss.
  • Interpretability: Attention maps provide a "window" into the model's decision-making process, allowing clinicians to verify if the AI is focusing on the correct anatomical features.

Future Challenges and Directions

Despite significant progress, several hurdles remain for clinical deployment. Most researchers currently use "off-the-shelf" attention modules, but there is a growing need for designs tailored to specific medical geometries, such as tubular blood vessels. Furthermore, Transformers remain memory-intensive, posing challenges for high-resolution 3D scans or real-time surgical guidance. Future research must also prioritize standardized evaluation protocols and rigorous failure analysis to ensure clinical safety and reliability in diverse real-world settings.

Summary updated

Abstract

Medical image segmentation plays an important role in computer-aided diagnosis. Attention mechanisms that distinguish important parts from irrelevant parts have been widely used in medical image segmentation tasks. This paper systematically reviews the basic principles of attention mechanisms and their applications in medical image segmentation. First, we review the basic concepts of attention mechanism and formulation. Second, we surveyed over 300 articles related to medical image segmentation, and divided them into two groups based on their attention mechanisms, non-Transformer attention and Transformer attention. In each group, we deeply analyze the attention mechanisms from three aspects based on the current literature work, i.e., the principle of the mechanism (what to use), implementation methods (how to use), and application tasks (where to use). We also thoroughly analyzed the advantages and limitations of their applications to different tasks. Finally, we summarize the current state of research and shortcomings in the field, and discuss the potential challenges in the future, including task specificity, robustness, standard evaluation, etc. We hope that this review can showcase the overall research context of traditional and Transformer attention methods, provide a clear reference for subsequent research, and inspire more advanced attention research, not only in medical image segmentation, but also in other image analysis scenarios.
BibTeX
@misc{xie2023attention,
  author = {Yutong Xie and Bing Yang and Qingbiao Guan and Jianpeng Zhang and Qi Wu and Yong Xia},
  title = {{Attention Mechanisms in Medical Image Segmentation: A Survey}},
  year = {2023},
  eprint = {2305.17937},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.17937},
}
Figure