This review paper provides a comprehensive look at how Deep Learning (DL) is transforming 2D-to-3D medical image registration—a critical task for modern surgery and radiotherapy. By analyzing 41 key studies, the authors map out the transition from slow, manual optimization to fast, automated neural networks.
TL;DR
2D-to-3D registration aligns real-time intraoperative images (like X-rays) with preoperative 3D scans (like CTs) to guide surgeons. This review highlights a shift toward millisecond-scale "direct regression" and hybrid architectures, while identifying a critical lack of uncertainty estimation and temporal modeling in current research.
What Problem Does This Paper Address?
In the operating room, surgeons often rely on 2D imaging (fluoroscopy, ultrasound, or endoscopy) because it is fast and low-dose. However, the high-resolution "map" they need for guidance is usually a 3D volume (CT or MRI) acquired days before.
The challenge is 2D-to-3D registration: finding the exact spatial transformation that aligns the 2D view with the 3D volume. This is "ill-posed" because a 2D image lacks depth information—it’s either a projection that flattens anatomy or a thin slice that ignores everything outside its plane. Traditional methods are slow and easily get "stuck" in wrong solutions if the initial guess isn't perfect. Deep learning offers a way to learn these complex spatial relationships offline, enabling robust, real-time alignment during surgery.
Methods
The authors categorize DL approaches across three main axes:
- Learning Paradigms:
- Supervised: Training on known transformations (often using synthetic data).
- Self-Supervised: Using image similarity metrics (like Normalized Cross-Correlation) as the training signal, often utilizing Differentiable Renderers (e.g., DiffDRR) to simulate X-rays from CT volumes within the neural network.
- Pipeline Design:
- Direct Regression: A single forward pass through a network to predict the transformation. It's incredibly fast (milliseconds) but can struggle with large misalignments.
- Iterative Refinement: The network predicts small corrections repeatedly, or a hybrid approach where a network provides a "warm start" for a classical optimizer.
- Architectures: While CNNs (like ResNet) remain the workhorse, the field is moving toward Hybrid CNN-Attention models and Vision Transformers to capture global anatomical context. The review also notes emerging interest in Implicit Neural Representations (INRs), which represent the transformation as a continuous function.
Datasets
The review highlights a significant data bottleneck. Because real-world "ground truth" (knowing exactly where a patient is relative to a scan during surgery) is hard to get, 30 out of 41 studies rely exclusively on synthetic data.
- Common Modalities: The vast majority of research focuses on X-ray-to-CT registration.
- Synthetic Generation: Researchers typically use CT volumes to render "Digitally Reconstructed Radiographs" (DRRs) to act as simulated X-rays.
- Clinical Data: Only a fraction of studies (14/41) evaluate their models on real clinical images, which often contain noise, surgical tools, and anatomical deformations not present in simulations.
A detailed breakdown of the studies, their architectures, and the anatomy targeted (ranging from spine to liver) can be found in the summary table:
Key Insights and Ideas
- The Explosion of Interest: The field is accelerating rapidly; 68% of the included studies were published between 2024 and mid-2026. Growth of the field by publication year
- Differentiable Physics: The integration of differentiable geometric operators (slicing and projection) directly into network architectures has enabled end-to-end self-supervised learning, bypassing the need for manual labels.
- The "Depth" Blind Spot: Errors measured in 2D (on the image plane) often hide massive errors in 3D (along the depth axis). The authors argue that 3D Mean Target Registration Error (mTRE) must remain the gold standard.
- The Reality Gap: Models trained on "clean" synthetic DRRs often fail on "messy" real X-rays. Domain adaptation (making synthetic data look real or vice versa) is becoming a core research pillar.
Key Results
- Speed: Direct regression models achieve inference times in the millisecond range, making them ideal for real-time surgical navigation.
- Accuracy: Iterative refinement networks generally outperform single-shot regression, especially when the initial misalignment is large.
- Dominance of X-ray: X-ray-to-CT remains the most mature application, while slice-to-volume registration (US/MRI) remains scarce and highly application-specific.
Limitations and Future Work
The authors identify two "gaps" that prevent these models from being truly clinical-ready:
- Uncertainty Estimation: None of the 41 studies provide a confidence score. In a clinical setting, a "wrong" registration is dangerous; the system must be able to say, "I'm not sure about this alignment."
- Temporal Neglect: Most models treat every frame of a video stream as an independent problem. The authors suggest using State-Space Models (like Mamba) to add temporal memory, allowing the system to use information from previous frames to improve current alignment without sacrificing speed.
- Reproducibility: Very few studies release their code or trained models, hindering the ability of the community to build upon existing work.
Summary updated