A thorough review on Deep Learning approaches to 2D-to-3D Registration in Medical Imaging

Andrea Borghesi, Julia van Beurden, Maarten van Alphen, Michiel van den Brekel, Luc Karssemakers, Jan-Jakob Sonke, Baris Karakullukcu, Jonas Teuwen, George YiasemisarXiv 2026 · arXiv

This review paper provides a comprehensive look at how Deep Learning (DL) is transforming 2D-to-3D medical image registration—a critical task for modern surgery and radiotherapy. By analyzing 41 key studies, the authors map out the transition from slow, manual optimization to fast, automated neural networks.

TL;DR

2D-to-3D registration aligns real-time intraoperative images (like X-rays) with preoperative 3D scans (like CTs) to guide surgeons. This review highlights a shift toward millisecond-scale "direct regression" and hybrid architectures, while identifying a critical lack of uncertainty estimation and temporal modeling in current research.

What Problem Does This Paper Address?

In the operating room, surgeons often rely on 2D imaging (fluoroscopy, ultrasound, or endoscopy) because it is fast and low-dose. However, the high-resolution "map" they need for guidance is usually a 3D volume (CT or MRI) acquired days before.

The challenge is 2D-to-3D registration: finding the exact spatial transformation that aligns the 2D view with the 3D volume. This is "ill-posed" because a 2D image lacks depth information—it’s either a projection that flattens anatomy or a thin slice that ignores everything outside its plane. Traditional methods are slow and easily get "stuck" in wrong solutions if the initial guess isn't perfect. Deep learning offers a way to learn these complex spatial relationships offline, enabling robust, real-time alignment during surgery.

Methods

The authors categorize DL approaches across three main axes:

  1. Learning Paradigms:
    • Supervised: Training on known transformations (often using synthetic data).
  2. Self-Supervised: Using image similarity metrics (like Normalized Cross-Correlation) as the training signal, often utilizing Differentiable Renderers (e.g., DiffDRR) to simulate X-rays from CT volumes within the neural network.
  3. Pipeline Design:
    • Direct Regression: A single forward pass through a network to predict the transformation. It's incredibly fast (milliseconds) but can struggle with large misalignments.
  4. Iterative Refinement: The network predicts small corrections repeatedly, or a hybrid approach where a network provides a "warm start" for a classical optimizer.
  5. Architectures: While CNNs (like ResNet) remain the workhorse, the field is moving toward Hybrid CNN-Attention models and Vision Transformers to capture global anatomical context. The review also notes emerging interest in Implicit Neural Representations (INRs), which represent the transformation as a continuous function.

Datasets

The review highlights a significant data bottleneck. Because real-world "ground truth" (knowing exactly where a patient is relative to a scan during surgery) is hard to get, 30 out of 41 studies rely exclusively on synthetic data.

  • Common Modalities: The vast majority of research focuses on X-ray-to-CT registration.
  • Synthetic Generation: Researchers typically use CT volumes to render "Digitally Reconstructed Radiographs" (DRRs) to act as simulated X-rays.
  • Clinical Data: Only a fraction of studies (14/41) evaluate their models on real clinical images, which often contain noise, surgical tools, and anatomical deformations not present in simulations.

A detailed breakdown of the studies, their architectures, and the anatomy targeted (ranging from spine to liver) can be found in the summary table:

Overview of included studies by architecture, task, and anatomy

Key Insights and Ideas

  • The Explosion of Interest: The field is accelerating rapidly; 68% of the included studies were published between 2024 and mid-2026. Growth of the field by publication year
  • Differentiable Physics: The integration of differentiable geometric operators (slicing and projection) directly into network architectures has enabled end-to-end self-supervised learning, bypassing the need for manual labels.
  • The "Depth" Blind Spot: Errors measured in 2D (on the image plane) often hide massive errors in 3D (along the depth axis). The authors argue that 3D Mean Target Registration Error (mTRE) must remain the gold standard.
  • The Reality Gap: Models trained on "clean" synthetic DRRs often fail on "messy" real X-rays. Domain adaptation (making synthetic data look real or vice versa) is becoming a core research pillar.
Figure 1. Number of included studies per publication year (n = 41). Sixty-eight per cent were published from 2024 onwards. ∗2026 covers January to June only, the end of the search window.

Key Results

  • Speed: Direct regression models achieve inference times in the millisecond range, making them ideal for real-time surgical navigation.
  • Accuracy: Iterative refinement networks generally outperform single-shot regression, especially when the initial misalignment is large.
  • Dominance of X-ray: X-ray-to-CT remains the most mature application, while slice-to-volume registration (US/MRI) remains scarce and highly application-specific.

Limitations and Future Work

The authors identify two "gaps" that prevent these models from being truly clinical-ready:

  1. Uncertainty Estimation: None of the 41 studies provide a confidence score. In a clinical setting, a "wrong" registration is dangerous; the system must be able to say, "I'm not sure about this alignment."
  2. Temporal Neglect: Most models treat every frame of a video stream as an independent problem. The authors suggest using State-Space Models (like Mamba) to add temporal memory, allowing the system to use information from previous frames to improve current alignment without sacrificing speed.
  3. Reproducibility: Very few studies release their code or trained models, hindering the ability of the community to build upon existing work.

Summary updated

Abstract

2D-to-3D image registration is a key component for several medical imaging purposes such as intraoperative guidance, enabling the alignment between preoperative 3D volumes and real-time 2D images. While conventional optimisation-based methods often struggle with non-convex objectives, sensitivity to initialisation, and high computational demands, deep learning (DL) has emerged as a promising alternative, offering improved alignment robustness and the potential for real-time performance. This review provides a comprehensive analysis of DL frameworks developed to address the challenges inherent in 2D-to-3D registration. We first outline how core concepts of DL are adapted to the registration problem. We then examine recent methodological advancements from multiple perspectives, including pipeline design, architectural innovations, and domain adaptation strategies. Particular attention is given to quantitative metrics, highlighting how performance is defined and measured across studies. Across the 41 included studies, we find that direct regression is the most common pipeline (23 studies, against 18 iterative ones), that hybrid CNN-attention architectures are gaining ground, and that research remains heavily concentrated on X-ray-to-CT registration. We conclude by identifying the absence of per-case uncertainty estimation and the strong neglect of temporal or sequential modelling as the two methodological gaps most consequential for clinical readiness, and discuss state-space models as a way to add temporal memory within intraoperative latency budgets. Further opportunities lie in more principled domain adaptation and in the integration of foundation models for representation learning.
BibTeX
@misc{borghesi2026thorough,
  author = {Andrea Borghesi and Julia van Beurden and Maarten van Alphen and Michiel van den Brekel and Luc Karssemakers and Jan-Jakob Sonke and Baris Karakullukcu and Jonas Teuwen and George Yiasemis},
  title = {{A thorough review on Deep Learning approaches to 2D-to-3D Registration in Medical Imaging}},
  year = {2026},
  eprint = {2611.17468},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17468},
}
Figure