Unsupervised Exemplar-Based Image-to-Image Translation and Cascaded Vision Transformers for Tagged and Untagged Cardiac Cine MRI Registration

Meng Ye, Mikael Kanski, Dong Yang, Leon Axel, Dimitris N. Metaxas

Introduction

Accurate assessment of cardiac function often requires integrating information from multiple MRI modalities. Untagged cine MRI (cMR) is the standard for evaluating global heart anatomy and function, while tagged cine MRI (tMR) employs a grid-like pattern to track myocardial strain and torsion. Analyzing tMR images typically requires segmenting the myocardium; however, the presence of tags obscures tissue boundaries, making manual or automated segmentation difficult. A common solution is multi-modal registration: warping a cMR image (where segmentation is easier) to match a corresponding tMR image.

This task is challenging due to the significant domain gap between modalities and the large deformations caused by cardiac motion and variations in breath-holding. This paper introduces ECaT, a novel two-stage framework that combines unsupervised, exemplar-based image translation with cascaded Vision Transformers (ViTs) to achieve state-of-the-art registration without requiring perfectly aligned training data.

Methodology: The ECaT Framework

The ECaT pipeline addresses the registration problem in two distinct phases: style-aware image translation followed by cascaded registration.

1. Style-Aware Image-to-Image Translation

To bridge the domain gap, the framework first translates the tMR image into a "fake cMR" image. Unlike standard translation models that learn a generic modality style, ECaT uses a specific cMR image as a style exemplar. This ensures that the translated image adopts the specific intensity distribution and contrast of the target scan. An unsupervised correspondence mechanism aligns features between the source tMR and the style exemplar without needing pre-aligned pairs.

2. Cascaded Vision Transformer Registration

Once the images are in the same "cMR-like" domain, the model performs mono-modal registration. ECaT utilizes Vision Transformers (ViTs), which leverage self-attention to capture long-range spatial dependencies, making them superior to traditional CNNs for predicting large deformations.

The ECaT architecture: (a) Overall pipeline showing translation and cascaded registration; (b) Exemplar-based I2I module; (c) TransUnet-based registration network; (d) ViT block details.

The registration is cascaded: it first predicts a global affine transformation (handling translation, rotation, and scaling) and then a local non-rigid deformation. Efficiency is maintained by sharing a "TransUnet" feature extractor between both stages, which reduces parameter count and prevents overfitting.

Experimental Results

The model was evaluated on a clinical dataset of 23 subjects, comprising 5,575 image pairs across training, validation, and testing sets.

Performance Metrics

ECaT demonstrated significant improvements in both translation and registration quality:

  • Translation: Achieved a Peak Signal-to-Noise Ratio (PSNR) of 27.516, outperforming baselines like CycleGAN (22.966) and RegGAN (25.207).
  • Registration: Reached a mean Dice score of 77.4%, surpassing traditional methods like MIND (72.6%) and deep learning models like VoxelMorph (72.0%).
  • Efficiency: The system is highly optimized for clinical use, with an inference time of approximately 0.068 seconds per pair.
Qualitative comparison of registration results. ECaT (h) achieves higher Dice scores and better alignment of myocardium boundaries compared to ablation variants (c-e) and baseline warped images.

Key Insights

The authors found that decoupling the translation and registration tasks is crucial. Training them simultaneously often leads to "trivial solutions" where the model fails to learn meaningful geometric transformations. Furthermore, using patient-specific exemplars for translation significantly improves the robustness of the downstream registration by normalizing appearance variations.

Conclusion and Future Work

The ECaT framework provides a robust, unsupervised solution for the difficult task of tMR-to-cMR registration. By combining style-aware translation with the long-range modeling capabilities of cascaded ViTs, it achieves high accuracy and clinical-grade speed. Future research will focus on extending this 2D approach to 3D cardiac volumes and exploring its applicability to other multi-modal registration tasks, such as CT-to-MRI.

Summary updated

Abstract

Multi-modal registration between tagged and untagged cardiac cine magnetic resonance (MR) images remains difficult, due to the domain gap and large deformations between the two modalities. Recent work using an image-to-image translation (I2I) module to overcome the domain gap can convert the multi-modal into a mono-modal registration task and take advantage of advanced mono-modal registration architectures. However, they often ignore two issues: the sample-specific style of each image to be registered during I2I and large hybrid rigid and non-rigid deformations between modalities. We first propose an exemplar-based I2I module capable of unsupervised cross-domain correspondence learning to enforce the style consistency between the fake image and the image to be registered. Then we propose an efficient cascaded vision transformer-based registration network to predict both the affine and non-rigid deformations, in which a single feature embedding subnetwork is shared by the two stages of deformation prediction. We validated our method on a clinical cardiac MR dataset with paired but unaligned untagged and tagged MR images. The results show that our method outperforms traditional methods significantly in terms of the I2I quality and multi-modal image registration accuracy.
BibTeX
@inproceedings{ye2024unsupervised,
  author = {Meng Ye and Mikael Kanski and Dong Yang and Leon Axel and Dimitris N. Metaxas},
  title = {{Unsupervised Exemplar-Based Image-to-Image Translation and Cascaded Vision Transformers for Tagged and Untagged Cardiac Cine MRI Registration}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00747},
}
Figure