Multiresolution Aggregation Transformer UNet Based on Multiscale Input and Coordinate Attention for Medical Image Segmentation

Shaolong Chen, Changzhen Qiu, Weiping Yang, Zhiyong Zhang2022 · DOI

TL;DR

This paper introduces MRA-TUNet, a novel deep learning architecture for medical image segmentation that effectively fuses multiresolution inputs and leverages coordinate attention. By enhancing feature extraction at both the encoder and decoder stages, MRA-TUNet achieves state-of-the-art performance on cardiac and atrial segmentation tasks, outperforming existing methods in dice score, precision, and recall.

What Problem Does This Paper Address?

Medical image segmentation is crucial for diagnosis, treatment planning, and disease monitoring. While UNet-based architectures and transformers have shown great promise, existing methods often struggle to comprehensively leverage multiscale information within medical images. This limitation hinders their ability to accurately capture both coarse anatomical structures and fine-grained details, especially in complex anatomical regions. Furthermore, the integration of multiresolution inputs and the exploitation of global and local feature dependencies remain areas with significant room for improvement.

Methods

The proposed MRA-TUNet builds upon the UNet architecture by incorporating several novel components:

  • Multiresolution Aggregation Module (MRAM): This module is applied at the encoder side to fuse input image information from different resolutions. It first extracts features from the current resolution image and then cascades them with features from the previous convolution unit. This cascaded feature is then processed by a coordinate attention mechanism before being passed through a convolution for enhanced feature extraction.
  • Convolutional Vision Transformer (CvT) as Encoder: Instead of a standard transformer encoder, the paper utilizes CvT, which integrates convolutional operations into the vision transformer. This approach aims to leverage the strengths of both CNNs (local feature extraction) and transformers (global context modeling).
  • Output Feature Selection Module (OFSM): At the decoder side, this module fuses features from the four decoder layers. It concatenates these features, extracts them through a convolution, and then applies coordinate attention for further refinement before the final segmentation prediction.
  • Coordinate Attention (CA): This attention mechanism is integrated into both the MRAM and OFSM. CA captures long-range dependencies by encoding spatial information into channel attention, allowing the network to focus on relevant features more effectively.

Datasets

The proposed MRA-TUNet was evaluated on two publicly available datasets:

  • Automated Cardiac Diagnosis Challenge (ACDC): This dataset consists of 100 3D cardiac MRI scans with ground truth annotations for the right ventricle (RV), myocardium (Myo), and left ventricle (LV). The data was split into training, validation, and testing sets with a 7:1:2 ratio.
  • 2018 Atrial Segmentation Challenge (2018 ASC): This dataset comprises 154 3D cardiac MRI scans with ground truth annotations for the left atrium (LA). Similar to ACDC, a 7:1:2 split was used for training, validation, and testing.

All image slices were normalized to a 0-1 range before being used for model training.

Key Insights and Ideas

  • Multiresolution Fusion is Key: The paper strongly advocates for the necessity of fusing information from different image resolutions. By treating high-resolution images for local features and low-resolution images for global features, and then effectively merging them, the model achieves better tissue localization.
  • Coordinate Attention Enhances Focus: The integration of coordinate attention in both the MRAM and OFSM modules proves to be effective in improving segmentation performance. This suggests that explicitly modeling spatial relationships and dependencies is beneficial for medical image segmentation.
  • CvT Balances Local and Global: Using CvT as the encoder strikes a good balance between capturing local image details (via convolutions) and understanding global context (via transformer mechanisms), which is crucial for complex anatomical structures.
  • Encoder-Decoder Feature Fusion Matters: The paper highlights the importance of feature fusion at both the encoder (MRAM) and decoder (OFSM) stages, demonstrating that a holistic approach to multiscale feature aggregation leads to superior results.

Key Results

MRA-TUNet demonstrated superior performance compared to eight state-of-the-art medical image segmentation methods on both the ACDC and 2018 ASC datasets.

  • ACDC Performance: Achieved average dice scores of 0.911 for RV, 0.890 for Myo, and 0.961 for LV.
  • 2018 ASC Performance: Achieved an average dice score of 0.923 for LA.
  • Overall Superiority: Outperformed other methods in dice score, precision, and recall across both datasets.
  • Qualitative Improvements: Visualizations showed that MRA-TUNet accurately segmented challenging structures like the right ventricle and myocardium, producing more complete and precise segmentations compared to other methods.

While the training time for MRA-TUNet is longer than some other methods due to its complex architecture and skip connections, the authors argue that this is acceptable for medical image segmentation where real-time performance is not always critical, and the performance gains justify the increased training duration.

Limitations and Future Work

The authors acknowledge a few limitations and suggest future research directions:

  • Shared Encoder for Multiresolution Inputs: The current architecture uses a shared encoder for multiresolution inputs. The authors suggest exploring a multibranch encoding network to potentially improve feature extraction by allowing specialized processing for different resolutions.
  • Encoder-Side Feature Fusion Only: The current method fuses features from different resolutions primarily at the encoder side. Future work could investigate feature fusion at the decoder side as well, to further enhance the integration of multiscale information.
  • Edge Segmentation Accuracy: While the method excels at tissue localization, the accuracy of segmenting the precise edges of structures is noted as an area that could be improved, possibly due to information loss during downsampling for low-resolution inputs.

Summary updated

BibTeX
@misc{chen2022multiresolution,
  author = {Shaolong Chen and Changzhen Qiu and Weiping Yang and Zhiyong Zhang},
  title = {{Multiresolution Aggregation Transformer UNet Based on Multiscale Input and Coordinate Attention for Medical Image Segmentation}},
  year = {2022},
  doi = {10.3390/s22103820},
}
Figure