Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion

Srikrishna Jaganathan, Maximilian Kukla, Jian Wang, Karthik Shetty, Andreas MaierWACV 2023

TL;DR

This paper introduces a self-supervised deep learning framework for aligning 2D X-ray images with 3D CT scans, crucial for image-guided medical interventions. By combining simulated X-ray generation with novel unsupervised domain adaptation techniques, the method overcomes the need for large, paired annotated datasets, achieving high accuracy and robustness on real-world X-ray images.

What Problem Does This Paper Address?

Accurate fusion of live fluoroscopic X-ray images with preoperative CT scans is vital for guiding minimally invasive medical procedures. This fusion relies on precise 2D/3D registration, which aligns the 3D CT volume with the patient's current position captured by the X-ray. Traditional optimization-based methods are sensitive to initialization and can struggle with variations in imaging conditions. While deep learning (DL) approaches have improved robustness and speed, they typically require extensive paired datasets (CT volumes with corresponding X-rays and ground truth registration), which are costly and difficult to obtain, especially for diverse anatomies and interventions. Generating simulated X-ray projections from CT volumes offers a way to create training data without real pairs, but this introduces a "domain gap" between simulated and real X-rays, leading to performance degradation. This paper aims to bridge this gap by developing a self-supervised registration method that leverages simulated data while adapting to real X-ray images without requiring any paired annotations.

Methods

The core of the proposed method is a self-supervised 2D/3D rigid registration framework built upon the Deep Iterative 2D/3D Registration Network (DIRN) [20]. The key innovations lie in overcoming the domain gap:

  • Simulated Training with Domain Randomization: The framework is initially trained using Digitally Reconstructed Radiographs (DRRs) generated from CT volumes. To improve robustness, these DRRs are rendered with various styles, including bone-only projections and realistic projections with added noise, contrast adjustments, and inversions. This "domain randomization" exposes the network to a wider range of simulated appearances.
  • Unsupervised Feature Space Domain Adaptation: To bridge the gap between simulated and real X-rays, the authors employ two unsupervised techniques:
  • Adversarial Feature Encoder (AFE): This component uses a feature discriminator to ensure that the feature representations extracted from real X-ray images and simulated DRRs are indistinguishable. This encourages the encoder to learn features that are invariant to the domain.
  • Barlow Twins (BT) [55]: This module further enforces style invariance by minimizing the cross-correlation between feature representations of different augmented versions of the simulated DRRs. This helps the network learn representations that are robust to variations in style. These two components are trained end-to-end with the DIRN, using shared encoder weights.

  • Unsupervised Pixel Space Domain Adaptation: To further enhance performance on real X-rays, a separate image-to-image style transfer network is used. The authors employ the Contrastive Unpaired Translation (CUT) [35] method, which learns to translate real X-ray images into the style of simulated DRRs. This network is trained independently and applied during inference to transform real X-rays into a style that the registration network has already encountered during its simulated training.

Datasets

The paper utilizes a clinical Cone Beam CT (CBCT) reconstruction dataset for training and evaluation. This dataset comprises X-ray images and corresponding 3D CBCT volumes from the thoracic and lumbar regions of 55 patients. For supervised training (used for comparison), random initial transformations were applied to create paired samples. For the self-supervised scenario, the dataset was adapted to use unpaired real X-ray images and simulated DRRs. The authors also mention using a dataset of 16,000 unpaired real X-ray and bone projection style DRR images for training the pixel space domain adaptation network and the adversarial feature encoder. The test set consists of 3600 samples from Anterior Posterior (AP) and lateral (LAT) views.

Key Insights and Ideas

  • Domain Adaptation is Crucial for Self-Supervised Medical Registration: The paper strongly demonstrates that simply training on simulated data is insufficient due to the domain gap. Effective unsupervised domain adaptation techniques are essential for achieving competitive performance on real-world medical images.
  • Combining Feature and Pixel Space Adaptation is Synergistic: The authors show that integrating both feature-level (AFE, BT) and pixel-level (CUT) domain adaptation leads to the best results, suggesting that addressing domain discrepancies at multiple levels is beneficial.
  • Self-Supervised Learning Can Approach Supervised Performance: The proposed method achieves a high success rate that is close to supervised methods, highlighting the potential of self-supervised learning to reduce reliance on expensive annotated data in medical imaging.
  • Barlow Twins for Style Invariance: The application of Barlow Twins, originally for representation learning, to enforce style invariance in feature representations for registration is a novel contribution.

Key Results

The proposed self-supervised framework achieves a registration accuracy of 1.83 ± 1.16 mm with a success ratio (SR) of 90.1% on real X-ray images. This represents a significant improvement of 23.9% in SR compared to other annotation-free algorithms.

  • Ablation Studies:
    • Simulated training alone (with domain randomization) achieved an SR of 66.2%.
  • Adding feature space domain adaptation improved SR to 72.2%.
  • Adding pixel space domain adaptation further boosted SR to 85.8%.
  • The combination of both feature and pixel space domain adaptation yielded the best result of 90.1% SR.
  • Comparison to State-of-the-Art: The method closely matches the performance of supervised DIRN [20] (SR 99.4%) while significantly outperforming optimization-based methods [52, 50] and the simulated baseline.

Limitations and Future Work

  • Registration Error: While the success ratio is high, the authors acknowledge that the registration error (mRPD) is higher compared to optimization-based and supervised methods. They suggest that an optimization-based refinement step could be applied if lower registration error is critical for specific applications.
  • Computational Cost: While the learned encoder weights are used during inference, the style transfer network for pixel space adaptation adds a computational overhead. However, this is described as a single forward pass.
  • Future Work: The paper implicitly suggests that further exploration of optimization-based refinement could be a future direction to achieve even lower registration errors. The success of the self-supervised approach also opens doors for applying similar techniques to other medical imaging registration tasks.

Summary updated

BibTeX
@inproceedings{jaganathan2023self,
  author = {Srikrishna Jaganathan and Maximilian Kukla and Jian Wang and Karthik Shetty and Andreas Maier},
  title = {{Self-Supervised 2D/3D Registration for X-Ray to CT Image Fusion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure