Few Shot Medical Image Segmentation with Cross Attention Transformer

Yi Lin, Yufan Chen, Kwang-Ting Cheng, Hao Chen

Introduction

Deep learning has significantly advanced medical image segmentation; however, these models typically require extensive datasets paired with precise, pixel-level annotations. In the clinical domain, obtaining such labels is a major bottleneck, as it requires significant time and expertise from radiologists, particularly for complex 3D modalities like CT and MRI.

Few-Shot Segmentation (FSS) addresses this challenge by enabling models to generalize to new classes (e.g., unseen organs or pathologies) using only a few labeled examples, known as the "support set." Despite their potential, traditional FSS methods often suffer from unidirectional information flow—transferring knowledge from the support to the query image while ignoring the inherent correlations between the two. This paper introduces CAT-Net, a framework utilizing a Cross Masked Attention Transformer to facilitate bidirectional communication and iterative refinement.

Methodology: The CAT-Net Architecture

CAT-Net is built upon three primary innovations designed to maximize the utility of limited data.

1. Mask Incorporated Feature Extraction (MIFE)

The process begins with a ResNet-50 backbone. Unlike standard feature extractors, MIFE integrates the support mask at an early stage. This ensures that the model prioritizes relevant anatomical features from the outset, filtering out irrelevant background noise before the data reaches the transformer modules.

2. Cross Masked Attention Transformer (CMAT)

The CMAT module is the core of the CAT-Net architecture. While standard transformers compute global attention, CMAT employs "masked attention" to focus exclusively on foreground information. This approach mines mutual correlations between support and query features. Crucially, CMAT enables a "mutual boosting" mechanism: query features are used to refine support features, and vice versa. This two-way dialogue creates a more robust representation of the target anatomy, compensating for variations in patient scans.

3. Iterative Refinement Framework

To achieve high-precision boundaries, CAT-Net employs an iterative refinement loop. The model's initial predictions are fed back into the CMAT module across multiple passes (optimally four). This process utilizes a double threshold strategy:

  • Strict Threshold: Used for loss calculation to ensure high-confidence learning.
  • Relaxed Threshold: Used to "dilate" the mask for the subsequent iteration, preventing the premature exclusion of valid foreground pixels.

Experimental Evaluation

The researchers evaluated CAT-Net on three diverse public datasets:

  • Abd-CT: 30 abdominal CT scans focusing on the liver, kidneys, and spleen.
  • Abd-MRI: 20 abdominal MRI scans focusing on the same organs.
  • Card-MRI: 35 cardiac MRI scans focusing on the left ventricular blood pool, myocardium, and right ventricle.

The model was tested using 5-fold cross-validation under two distinct settings to validate its robustness against variations in the support set.

Results and Key Insights

CAT-Net established new state-of-the-art (SOTA) benchmarks, consistently outperforming existing methods such as ALP-Net and Q-Net.

  • Performance Metrics: The model achieved Dice scores of 66.59% on Abd-CT (a 1.76% improvement over previous SOTA), 75.18% on Abd-MRI, and 79.03% on Card-MRI.
  • Ablation Studies: Experiments confirmed that the "mutual boosting" mechanism is critical. Reverting to a unidirectional information flow resulted in a significant performance decline, highlighting the importance of bidirectional cross-attention.
  • Iterative Success: The "coarse-to-fine" approach proved effective, with each iteration progressively sharpening the segmentation boundaries.

Discussion and Future Directions

While CAT-Net represents a significant advancement in FSS for medical imaging, several avenues for future research remain. Currently, the model processes 3D volumes as 2D slices; extending the architecture to native 3D networks could better capture volumetric spatial dependencies. Additionally, while the study focused on healthy organs, future iterations will need to address the complexities of pathological tissues and anatomical malformations. Finally, optimizing the computational efficiency of the iterative refinement process will be essential for integration into real-time clinical workflows.

Conclusion

CAT-Net demonstrates that bidirectional feature interaction and iterative refinement can significantly improve the accuracy of few-shot medical image segmentation. By focusing on mutual correlations between support and query images, the model reduces the dependency on large-scale annotated datasets, moving a step closer to scalable, AI-assisted clinical diagnostics.

Summary updated

BibTeX
@inproceedings{lin2023few,
  author = {Yi Lin and Yufan Chen and Kwang-Ting Cheng and Hao Chen},
  title = {{Few Shot Medical Image Segmentation with Cross Attention Transformer}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2023},
  doi = {10.1007/978-3-031-43895-0\_22},
}
Figure