DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation

Zhehao Wang, Xian Lin, Nannan Wu, Li Yu, Kwang-Ting Cheng, Zengqiang YanAAAI 2024 · DOI

Introduction

Transformer architectures have revolutionized computer vision by enabling models to capture long-range dependencies through self-attention. However, their application in medical image segmentation is hampered by high computational costs and a phenomenon known as "attention collapse." DTMFormer (Dynamic Token Merging Transformer) addresses these issues by introducing a plug-and-play module that dynamically merges redundant tokens into a smaller set of semantic representatives. This approach not only reduces the $O(n^2)$ complexity of self-attention but also improves segmentation accuracy by focusing the model on critical image regions.

The Challenge: Attention Collapse and Redundancy

Medical images, such as CT or MRI scans, often contain large regions of homogeneous texture (e.g., the interior of an organ or background). Calculating pairwise dependencies for every pixel in these redundant areas is computationally wasteful. Furthermore, in many hybrid CNN-Transformer models, deeper transformer layers often suffer from "attention collapse," where the model assigns nearly identical dependency scores to all patches, effectively rendering the transformer module a useless bypass.

Attention maps in TransUnet and FATNet across different layers, demonstrating the loss of discriminative attention in deeper stages.

This collapse is driven by two factors: the high model complexity relative to the size of typical medical datasets and the inherent noise introduced by calculating attention over redundant, non-informative regions.

Methodology: DTMFormer

The DTMFormer replaces standard transformer blocks with a two-stage architecture designed for efficiency and precision. It consists of two primary components:

1. Attention-guided Token Merging (ATM)

Instead of processing every image patch independently, the ATM module clusters patches into "semantic tokens." It identifies "center" tokens based on feature similarity and dependence importance (using existing attention maps to prioritize boundaries). Non-center tokens are then fused into these centers. Unlike pruning methods that discard data, DTMFormer merges it, preserving abstract background context while reducing the total token count.

Visualization of token clustering, where pixels with similar semantic features are grouped together to streamline computation.

2. Light Token Reconstruction (LTR)

To maintain the high-resolution output required for medical segmentation, the LTR module maps compressed semantic features back to their original spatial locations. By utilizing the merging history from the ATM phase and lightweight linear layers, LTR ensures pixel-perfect reconstruction with minimal overhead.

Experimental Results

DTMFormer was evaluated on three major benchmarks: ACDC (cardiac MRI), ISIC (skin lesions), and BTCV (abdominal CT).

  • Performance Gains: On the ACDC dataset, the FATNet+DTMFormer configuration achieved a Dice score of 92.53%, surpassing several state-of-the-art 3D models.
  • Efficiency: Integrating DTMFormer into the SETR baseline reduced the parameter count by 18 million and slashed GFLOPs from 53.2 to 9.8, representing a significant reduction in computational footprint without sacrificing performance.
  • Qualitative Improvements: Visual analysis indicates that DTMFormer produces cleaner segmentation masks with fewer false positives in background regions compared to standard transformer-based models.

Conclusion

DTMFormer provides a versatile, "plug-and-play" solution for enhancing transformer-based medical image segmentation. By treating redundancy as an opportunity for compression rather than a source of noise, it mitigates attention collapse and significantly lowers computational barriers. Future work will focus on optimizing the reconstruction process and extending the framework to other modalities such as ultrasound and pathology.

Summary updated

BibTeX
@inproceedings{wang2024dtmformer,
  author = {Zhehao Wang and Xian Lin and Nannan Wu and Li Yu and Kwang-Ting Cheng and Zengqiang Yan},
  title = {{DTMFormer: Dynamic Token Merging for Boosting Transformer-Based Medical Image Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i6.28394},
}
Figure