Introduction
Transformer architectures have revolutionized computer vision by enabling models to capture long-range dependencies through self-attention. However, their application in medical image segmentation is hampered by high computational costs and a phenomenon known as "attention collapse." DTMFormer (Dynamic Token Merging Transformer) addresses these issues by introducing a plug-and-play module that dynamically merges redundant tokens into a smaller set of semantic representatives. This approach not only reduces the $O(n^2)$ complexity of self-attention but also improves segmentation accuracy by focusing the model on critical image regions.
The Challenge: Attention Collapse and Redundancy
Medical images, such as CT or MRI scans, often contain large regions of homogeneous texture (e.g., the interior of an organ or background). Calculating pairwise dependencies for every pixel in these redundant areas is computationally wasteful. Furthermore, in many hybrid CNN-Transformer models, deeper transformer layers often suffer from "attention collapse," where the model assigns nearly identical dependency scores to all patches, effectively rendering the transformer module a useless bypass.
This collapse is driven by two factors: the high model complexity relative to the size of typical medical datasets and the inherent noise introduced by calculating attention over redundant, non-informative regions.
Methodology: DTMFormer
The DTMFormer replaces standard transformer blocks with a two-stage architecture designed for efficiency and precision. It consists of two primary components:
1. Attention-guided Token Merging (ATM)
Instead of processing every image patch independently, the ATM module clusters patches into "semantic tokens." It identifies "center" tokens based on feature similarity and dependence importance (using existing attention maps to prioritize boundaries). Non-center tokens are then fused into these centers. Unlike pruning methods that discard data, DTMFormer merges it, preserving abstract background context while reducing the total token count.
2. Light Token Reconstruction (LTR)
To maintain the high-resolution output required for medical segmentation, the LTR module maps compressed semantic features back to their original spatial locations. By utilizing the merging history from the ATM phase and lightweight linear layers, LTR ensures pixel-perfect reconstruction with minimal overhead.
Experimental Results
DTMFormer was evaluated on three major benchmarks: ACDC (cardiac MRI), ISIC (skin lesions), and BTCV (abdominal CT).
- Performance Gains: On the ACDC dataset, the FATNet+DTMFormer configuration achieved a Dice score of 92.53%, surpassing several state-of-the-art 3D models.
- Efficiency: Integrating DTMFormer into the SETR baseline reduced the parameter count by 18 million and slashed GFLOPs from 53.2 to 9.8, representing a significant reduction in computational footprint without sacrificing performance.
- Qualitative Improvements: Visual analysis indicates that DTMFormer produces cleaner segmentation masks with fewer false positives in background regions compared to standard transformer-based models.
Conclusion
DTMFormer provides a versatile, "plug-and-play" solution for enhancing transformer-based medical image segmentation. By treating redundancy as an opportunity for compression rather than a source of noise, it mitigates attention collapse and significantly lowers computational barriers. Future work will focus on optimizing the reconstruction process and extending the framework to other modalities such as ultrasound and pathology.
Summary updated