Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis

Vu Minh Hieu Phan, Yutong Xie, Bowen Zhang, Yuankai Qi, Zhibin Liao, Antonios Perperidis, Son Lam Phung, Johan W. Verjans, Minh-Son To

Abstract

Researchers have developed UNest (UNet Structural Transformer), a new architecture that significantly improves how AI synthesizes medical images (like turning an MRI into a CT scan) when paired training data isn't available. By using the Segment-Anything Model (SAM) to guide "Structural Attention," the model learns to prioritize anatomical accuracy, reducing synthesis errors by up to 19.3% compared to previous state-of-the-art methods.

Introduction

Medical image synthesis—generating one modality (e.g., CT) from another (e.g., MRI)—is vital for reducing patient radiation exposure and costs. However, obtaining "paired" data (identical scans of the same patient in two modalities) is difficult due to patient movement and different scanner resolutions. While Vision Transformers (ViTs) are great at capturing long-range relationships, they often struggle with "unpaired" synthesis. Because they lack the "inductive bias" found in traditional convolutional neural networks, ViTs often fail to reconstruct intricate anatomical structures, tending to focus on irrelevant background noise instead.

Methodology: The UNest Framework

The authors introduce UNest, which injects structural knowledge back into the Transformer architecture through three key innovations:

  1. Structural Attention Block (ST): Instead of looking at the whole image uniformly, the ST block segregates the image into foreground (anatomy) and background.
  2. Dual-Attention Strategy: The foreground uses "Structural Attention" to aggregate context specifically within anatomical regions, while the background uses "Local Attention" to handle surrounding areas efficiently.
  3. SAM-Guided Training: To define the "foreground," the researchers used the Segment-Anything Model (SAM) to automatically extract masks of the primary anatomy. This provides a "structural hint" during training without requiring manual labeling by radiologists.
The UNest architecture featuring the Structural Transformer (ST) block and its integration within a CycleGAN framework for unpaired image synthesis.

The entire system is wrapped in a CycleGAN framework, which uses two generators and two discriminators to ensure that an image translated from one domain to another can be accurately translated back.

Experimental Results

The model was evaluated on two major public datasets: MRXFDG (neuroimaging) and AutoPET (whole-body scans). UNest outperformed existing methods like ResViT, UNETR, and Swin-UNETR across six different synthesis tasks. On the MRXFDG dataset, UNest improved the Mean Absolute Error (MAE) by 19.30% over Swin-UNETR.

Comparison of MRI-to-CT synthesis results and attention maps between UNETR and UNest. UNest focuses more precisely on anatomical structures.

Qualitative results show that UNest is much better at preserving "structural cavities" (like the sinuses) and complex joints (like the hips) which other models often "smear" or delete. These improvements were consistent across multiple runs and confirmed via statistical t-tests (p < 0.05).

Discussion: The "Global" Weakness

A key insight of the paper is that global attention (looking at every pixel at once) is actually a liability in unpaired medical imaging. Without a ground-truth target image to correct it, standard Transformers often focus on the air around a patient's head rather than the brain itself. By forcing the model to treat the anatomy and the background differently, UNest re-introduces the "spatial common sense" that pure Transformers typically lack.

Conclusion

UNest represents a significant step forward in medical image synthesis by leveraging foundation models like SAM to provide anatomical supervision in an unsupervised task. While the method relies on mask quality and adds some computational complexity, its ability to maintain anatomical boundaries is critical for clinical applications. Future work may extend this approach to 3D synthesis or lesion detection.

Summary updated

Abstract

Unpaired medical image synthesis aims to provide complementary information for an accurate clinical diagnostics, and address challenges in obtaining aligned multi-modal medical scans. Transformer-based models excel in imaging translation tasks thanks to their ability to capture long-range dependencies. Although effective in supervised training, their performance falters in unpaired image synthesis, particularly in synthesizing structural details. This paper empirically demonstrates that, lacking strong inductive biases, Transformer can converge to non-optimal solutions in the absence of paired data. To address this, we introduce UNet Structured Transformer (UNest) - a novel architecture incorporating structural inductive biases for unpaired medical image synthesis. We leverage the foundational Segment-Anything Model to precisely extract the foreground structure and perform structural attention within the main anatomy. This guides the model to learn key anatomical regions, thus improving structural synthesis under the lack of supervision in unpaired training. Evaluated on two public datasets, spanning three modalities, i.e., MR, CT, and PET, UNest improves recent methods by up to 19.30% across six medical image synthesis tasks. Our code is released at https://github.com/HieuPhan33/MICCAI2024-UNest.
BibTeX
@inproceedings{phan2024structural,
  author = {Vu Minh Hieu Phan and Yutong Xie and Bowen Zhang and Yuankai Qi and Zhibin Liao and Antonios Perperidis and Son Lam Phung and Johan W. Verjans and Minh-Son To},
  title = {{Structural Attention: Rethinking Transformer for Unpaired Medical Image Synthesis}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2024},
  url = {https://openreview.net/forum?id=oP5Jv7rlTA},
}
Figure