Improving Pelvic MR-CT Image Alignment with Self-Supervised Reference-Augmented Pseudo-CT Generation Framework

Daniel Kim, Mohammed A. Al-masni, Jaehun Lee, Dong-Hyun Kim, Kanghyun RyuWACV 2025

TL;DR

This paper introduces RegistFormer, a novel framework for generating aligned pseudo-CT images from misaligned MR-CT pairs. By leveraging a reference CT image and a deformation-aware attention mechanism, RegistFormer significantly improves structural alignment and texture preservation compared to existing methods, paving the way for more accurate downstream tasks like segmentation.

What Problem Does This Paper Address?

Accurate alignment of Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) scans is crucial for many medical applications, particularly in prostate cancer radiation therapy where CT is used for dose planning and MRI for precise organ delineation. However, aligning these two modalities is challenging due to significant differences in image appearance and shape variations caused by factors like patient positioning and organ filling. Traditional intensity-based registration methods often struggle with these complexities. While image-to-image (I2I) translation methods can generate synthetic CT images, they are prone to hallucinations and inaccuracies, especially when dealing with misaligned input data. This paper addresses the need for a robust and accurate method to generate aligned pseudo-CT images from misaligned MR-CT pairs, eliminating the requirement for perfectly aligned training data.

Methods

RegistFormer employs a two-stage framework:

  1. Initial Synthesis (Stage 1): This stage uses a synthesis network inspired by multimodal unsupervised image-to-image translation (MUNIT). It takes a misaligned MR image and a reference CT image as input. The network disentangles content (structure) from style (texture) and uses the MR content and CT style to generate an initial pseudo-CT. This stage aims to incorporate global style information from the reference CT while preserving the structural alignment with the input MR. Adaptive Instance Normalization (AdaIN) is used to transfer the CT's style.
  2. Local Refinement with Feature Aggregation (Stage 2): This stage refines the initial pseudo-CT by incorporating local features from the reference CT. The core innovation here is the Deformation-Aware Cross-Attention (DACA) block. DACA uses a pre-trained registration network to estimate a deformation field, which guides a windowed cross-attention mechanism. This mechanism aggregates spatially matched features from the reference CT into the MR's coordinate system, effectively correcting for misalignments and improving accuracy. The DACA block operates within a multi-resolution feature space of a U-Net architecture.

The framework is trained in a self-supervised manner using a novel combination of loss functions:

  • Structure-Preserving Loss (LpatchNCE): Based on contrastive learning, this loss ensures structural alignment with the input MR image by maximizing similarity between feature vectors of the generated pseudo-CT and the MR, while minimizing similarity with features from other locations. It uses a pre-trained VGG-19 network for feature extraction.
  • Misalignment-Robust Loss (Lcx): This contextual loss is used to preserve the texture of the reference CT image. It is designed to be resilient to misalignment, which is crucial given the dataset's nature.
  • Adversarial Loss (Lgan): A patch-based Least Squares GAN (LSGAN) is used to enhance image quality and realism.

Datasets

The primary dataset used is the SynthRAD2023 MR-CT pelvis pair dataset. This dataset consists of misaligned MR and CT scans from 165 patients. The authors pre-processed the MR images with N4 bias field correction and Nyul histogram matching, and CT images with windowed HU clipping and z-score normalization. For training, they used 112 patients, 18 for validation, and 35 for testing. Images were resized and random patch cropping (96x96) was used during training.

Key Insights and Ideas

  • Leveraging Misaligned Data: The most significant insight is the ability to effectively train a high-quality pseudo-CT generation model using misaligned MR-CT pairs. This bypasses the need for expensive and difficult manual alignment or perfectly paired datasets.
  • Deformation-Aware Attention: The DACA block is a clever way to integrate explicit deformation information into a cross-attention mechanism. This allows the model to "look" at the correct corresponding locations in the reference CT, even when the initial alignment is imperfect.
  • Two-Stage Refinement: The staged approach, starting with global style transfer and then performing local feature aggregation guided by deformation, provides a robust pipeline for both overall image synthesis and fine-grained alignment.
  • Self-Supervised Learning with Contextual Loss: The use of contextual loss (Lcx) is key to preserving CT texture despite misalignment, which is a common failure mode for other methods.

Key Results

  • Superior Alignment: RegistFormer achieved the second-best performance across multiple metrics, including Gradient Correlation (GC) and Normalized Mutual Information (NMI), outperforming many state-of-the-art registration and I2I methods.
  • Improved Texture and Sharpness: The method demonstrated good performance in terms of FID (lower is better, indicating better texture quality) and sharpness.
  • Downstream Task Performance: Crucially, the pseudo-CTs generated by RegistFormer led to significantly improved segmentation results, particularly for the urinary bladder, when used with a tool like TotalSegmentator. This highlights the clinical utility of the generated aligned images.
  • Ablation Studies: Both the deformation field and the local patch size in the DACA block were shown to be critical for performance. The combination of LpatchNCE and Lcx loss functions yielded the best results.

Limitations and Future Work

  • 2D Limitation: The current implementation operates on 2D slices due to GPU memory constraints with cross-attention. The authors plan to extend the framework to 3D using memory-efficient transformers.
  • Indirect Evaluation: Due to the inherent misalignment of the dataset, direct ground truth for alignment is unavailable. The authors relied on indirect metrics like GC and NMI and plan to validate with well-aligned datasets (e.g., full-body phantoms) in the future.
  • Potential for Further Improvement: The authors suggest using the generated pseudo-CTs for improving downstream MR-to-CT synthesis tasks where precise alignment is difficult. They also aim to explore full-body MR-CT alignment.

Summary updated

BibTeX
@inproceedings{kim2025improving,
  author = {Daniel Kim and Mohammed A. Al-masni and Jaehun Lee and Dong-Hyun Kim and Kanghyun Ryu},
  title = {{Improving Pelvic MR-CT Image Alignment with Self-Supervised Reference-Augmented Pseudo-CT Generation Framework}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure