Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging

Bernhard Stimpel, Christopher Syben, Tobias Würfl, Katharina Breininger, Philipp Hoelter, Arnd Dörfler, Andreas MaierarXiv 2019 · arXiv

TL;DR

This paper introduces a deep learning approach to translate MR projection images into synthetic X-ray projections for real-time hybrid image-guided interventions. By shifting network capacity to high-resolution layers and incorporating an edge-weighted loss function, the method achieves sharp, anatomically accurate X-ray synthesis with only a 6% deviation from ground truth.

What Problem Does This Paper Address?

In image-guided interventional procedures (like surgeries or catheter placements), X-ray fluoroscopy is the gold standard because it offers high spatial and temporal resolution. However, X-rays are notoriously poor at visualizing soft tissue. Magnetic Resonance Imaging (MRI) provides excellent soft-tissue contrast but is traditionally too slow for real-time intraoperative guidance.

Hybrid X-ray/MR systems aim to combine the strengths of both modalities. To effectively fuse or enhance these images, the data must exist in the same domain. Translating 2D MR projections directly into 2D X-ray projections, however, is an "ill-posed" problem. Dense structures like bone dominate X-ray images but yield virtually no signal in MRI. Furthermore, because projection imaging flattens 3D structures into a 2D plane, overlapping anatomy creates severe signal ambiguities that cannot be resolved by pixel intensity alone.

Methods

The authors developed a deep learning-based image-to-image translation framework using a fully convolutional generator network trained in a supervised manner. They introduced two key innovations to standard conditional Generative Adversarial Network (cGAN) architectures:

  1. High-Resolution Capacity Shift: Standard encoder-decoder networks (like Pix2Pix) typically place the majority of their residual blocks at the lowest-resolution bottleneck layer to capture global context. However, because the global outline in medical projections is already consistent between MR and X-ray, the authors redistributed the residual blocks to the higher-resolution layers to maximize the network's capacity for synthesizing fine details.
  2. High-Frequency Loss Weighting: To prevent the network from producing overly smooth or blurry images, the authors applied a Sobel filter to the ground-truth X-ray images to generate an edge map. This map acts as a pixel-wise weight during training, forcing the adversarial and feature-matching (VGG-19) loss functions to heavily prioritize sharp contours and edges over large, homogeneous regions.
Figure 1. A schematic visualization of the proposed network architecture. The numbers on the layers denote the feature dimensions and the convolution kernel size.

Datasets

Because simultaneous real-world MR and X-ray projection datasets are not yet available in clinical routine, the authors simulated a dataset using 3D CT and MR head scans from 13 patients.

The 3D volumes were rigidly registered using 3D Slicer and forward-projected using the CONRAD framework to simulate matching cone-beam X-ray and MR projection pairs. To ensure data diversity, they generated 450 projections per patient across a 360° rotation in the transversal plane, varying both azimuthal and inclination angles.

Figure 2. A schematic overview of the data generation process. Note that ultimately the goal is to acquire simultaneous projection images in hybrid X-ray and MR imaging (cf. Section 2.1). To generate training data for the underlying work, simulation was necessary.

Key Insights and Ideas

  • Bottlenecks Aren't Always Best: In natural image synthesis, the bottleneck layer is crucial for figuring out what to draw. In medical projection translation, the what (the patient's anatomy) is already structurally aligned. The real challenge is how to draw the fine details, making high-resolution layers a much more valuable allocation of network capacity.
  • The 10% That Matters: Homogeneous regions (like background air or uniform tissue) make up roughly 90% of a medical projection image. Without edge-weighting, the loss calculated from these uniform areas completely overwhelms the network. Prioritizing the remaining ~10% of pixels that contain edges drastically improves visual sharpness and eliminates erroneously synthesized details.

Key Results

  • Quantitative Accuracy: The proposed architecture combined with the edge-weighted loss function achieved a Mean Absolute Error (MAE) of 5.7% ± 0.9%, marking a 25% improvement over the baseline reference architecture.
  • Structural Fidelity: The model achieved a Structural Similarity Index (SSIM) of 0.913 ± 0.005 and a Peak Signal-to-Noise Ratio (PSNR) of 21.859 ± 0.676.
  • Real-Time Speed: Running unoptimized on an Nvidia Tesla V100 GPU, the network achieved an inference speed of ~24 frames per second, proving its viability for real-time interventional use.

Limitations and Future Work

  • Blind Spots for Missing Info: The network cannot synthesize features completely absent from the input MR signal. For instance, certain medical hardware (like ventricular shunts) or specific contrasted blood vessels do not show up in standard MR protocols and therefore cannot be translated.
  • Anatomical Variety: The study was limited to a small dataset of 13 head scans.
  • Future Directions: The authors plan to test the method on more complex body regions, such as the chest, where moving organs (the heart) and highly contrasting structures (ribs vs. lungs) present tougher challenges. They also aim to incorporate temporal consistency to smoothly process continuous fluoroscopic video streams.

Summary updated

Abstract

Hybrid X-ray and magnetic resonance (MR) imaging promises large potential in interventional medical imaging applications due to the broad variety of contrast of MRI combined with fast imaging of X-ray-based modalities. To fully utilize the potential of the vast amount of existing image enhancement techniques, the corresponding information from both modalities must be present in the same domain. For image-guided interventional procedures, X-ray fluoroscopy has proven to be the modality of choice. Synthesizing one modality from another in this case is an ill-posed problem due to ambiguous signal and overlapping structures in projective geometry. To take on these challenges, we present a learning-based solution to MR to X-ray projection-to-projection translation. We propose an image generator network that focuses on high representation capacity in higher resolution layers to allow for accurate synthesis of fine details in the projection images. Additionally, a weighting scheme in the loss computation that favors high-frequency structures is proposed to focus on the important details and contours in projection imaging. The proposed extensions prove valuable in generating X-ray projection images with natural appearance. Our approach achieves a deviation from the ground truth of only $6$% and structural similarity measure of $0.913\,\pm\,0.005$. In particular the high frequency weighting assists in generating projection images with sharp appearance and reduces erroneously synthesized fine details.
BibTeX
@misc{stimpel2019projection,
  author = {Bernhard Stimpel and Christopher Syben and Tobias Würfl and Katharina Breininger and Philipp Hoelter and Arnd Dörfler and Andreas Maier},
  title = {{Projection-to-Projection Translation for Hybrid X-ray and Magnetic Resonance Imaging}},
  year = {2019},
  eprint = {1911.08163},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.08163},
}

Loading related content...

Figure