The Impact of Machine Learning on 2D/3D Registration for Image-guided\n Interventions: A Systematic Review and Perspective

Mathias Unberath, Gao Cong, Hu Yi-cheng, Max Judish, Russell H. Taylor, Mehran Armand, Robert B. GrupparXiv 2021 · DOI · arXiv

Abstract

Image-based navigation is widely considered the next frontier of minimally invasive surgery, promising increased access to reproducible, safe, and high-precision procedures. A critical component of this navigation is 2D/3D registration—estimating the spatial relationship between 3D structures (e.g., CT scans) and 2D intraoperative images (e.g., fluoroscopy). While a mature technique, its clinical transition has been hindered by challenges like optimization brittleness and limited single-view performance. This review examines how machine learning (ML) approximates complex functional mappings to solve these notorious challenges, offering a systematic summary of recent advances and a perspective on future needs.

Introduction

In image-guided interventions, surgeons require precise spatial mapping to understand tool locations relative to patient anatomy. 2D/3D registration serves as the "GPS" for these procedures. Traditionally, this was solved via intensity-based optimization—an iterative process prone to getting stuck in local minima if not initialized perfectly. Furthermore, estimating 3D depth from a single 2D projection is mathematically ill-posed, and traditional methods often struggle with inconsistencies like surgical hardware appearing in the 2D view but not the 3D model.

Methodology

To assess the impact of machine learning on this field, a systematic review was conducted following a rigorous selection process. The authors analyzed the literature to categorize the functional roles ML plays in enhancing registration workflows.

PRISMA flow diagram illustrating the systematic search and selection process that resulted in 48 included studies.

A Taxonomy of Machine Learning Approaches

The review identifies five key functional themes where ML has reshaped 2D/3D registration:

  • Contextualization: Utilizing CNNs for landmark detection or anatomy segmentation to automate the "starting guess" (initialization) and mask irrelevant image features.
  • Representation Learning: Compressing complex 3D deformations into low-dimensional latent spaces (e.g., via autoencoders), simplifying the optimization landscape.
  • Similarity Modeling: Replacing traditional mathematical formulas with learned similarity metrics that offer a wider "capture range" and smoother optimization surfaces.
  • Direct Parameter Regression: Employing deep learning to predict 3D poses directly from 2D images in a single forward pass, enabling real-time potential.
  • Verification: Training models for quality control to predict registration success or provide uncertainty estimates.

The Data Challenge: Simulation vs. Reality

A significant finding of this review is the field's heavy reliance on synthetic data. Of the 48 studies, 32 relied exclusively on Digitally Reconstructed Radiographs (DRRs) simulated from CT volumes. This creates a "synthetic trap" where models fail to generalize to real clinical environments due to domain shift—the difference between perfect simulations and noisy, high-scatter clinical X-rays.

Clinical Perspective and Future Directions

While ML has improved registration speed and automation, most research focuses on single-object registration. Real-world surgical scenarios are far more complex, involving multiple moving bone fragments, surgical robots, and implants.

Clinical workflows for pelvic osteotomy and femoroplasty, highlighting the complexity of multi-object registration and the need for biomechanical validation.

To bridge the "bench-to-bedside" gap, the community must prioritize:

  1. Standardized Benchmarks: Large-scale public datasets with ground-truth 3D poses.
  2. Robustness: Techniques to handle domain shift between simulation and the OR.
  3. Compositional Models: Frameworks capable of simultaneous multi-body tracking.
  4. Uncertainty Quantification: Systems that can signal when an alignment is unreliable, which is critical for surgeon trust in autonomous workflows.

Conclusion

Machine learning has significantly advanced the state of 2D/3D registration, particularly in automation and capture range. However, the transition to routine clinical use requires a shift from synthetic benchmarks to robust, multi-object frameworks validated within real-world surgical workflows.

Summary updated

Abstract

Image-based navigation is widely considered the next frontier of minimally\ninvasive surgery. It is believed that image-based navigation will increase the\naccess to reproducible, safe, and high-precision surgery as it may then be\nperformed at acceptable costs and effort. This is because image-based\ntechniques avoid the need of specialized equipment and seamlessly integrate\nwith contemporary workflows. Further, it is expected that image-based\nnavigation will play a major role in enabling mixed reality environments and\nautonomous, robotic workflows. A critical component of image guidance is 2D/3D\nregistration, a technique to estimate the spatial relationships between 3D\nstructures, e.g., volumetric imagery or tool models, and 2D images thereof,\nsuch as fluoroscopy or endoscopy. While image-based 2D/3D registration is a\nmature technique, its transition from the bench to the bedside has been\nrestrained by well-known challenges, including brittleness of the optimization\nobjective, hyperparameter selection, and initialization, difficulties around\ninconsistencies or multiple objects, and limited single-view performance. One\nreason these challenges persist today is that analytical solutions are likely\ninadequate considering the complexity, variability, and high-dimensionality of\ngeneric 2D/3D registration problems. The recent advent of machine\nlearning-based approaches to imaging problems that, rather than specifying the\ndesired functional mapping, approximate it using highly expressive parametric\nmodels holds promise for solving some of the notorious challenges in 2D/3D\nregistration. In this manuscript, we review the impact of machine learning on\n2D/3D registration to systematically summarize the recent advances made by\nintroduction of this novel technology. Grounded in these insights, we then\noffer our perspective on the most pressing needs, significant open problems,\nand possible next steps.\n
BibTeX
@misc{unberath2021impact,
  author = {Mathias Unberath and Gao Cong and HU Yi-cheng and Max Judish and Russell H. Taylor and Mehran Armand and Robert B. Grupp},
  title = {{The Impact of Machine Learning on 2D/3D Registration for Image-guided\textbackslash\{\}n Interventions: A Systematic Review and Perspective}},
  year = {2021},
  eprint = {2108.02238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2108.02238},
}
Figure