Markerless Surgical Navigation and Motion Tracking

This direction is merged into another direction: marker-based tracking limitations (hardware-dependent spatial tracking). Its last version stays readable and citable here.

Version 1 of 150 papers52 conceptsfrozen 2026-10-04

Traditional computer-assisted surgery is a trade-off between precision and "clinical friction." To achieve sub-millimeter accuracy, surgeons have historically relied on extrinsic hardware: rigid frames bolted to bone, adhesive reflective markers on the skin, and bulky infrared camera arrays that require a constant, unobstructed line-of-sight. These systems clutter the operating room, extend setup times, and often require additional invasive incisions just to mount the tracking hardware.

A growing body of research is moving toward markerless navigation, where the system perceives the surgical environment using only natural features and integrated sensors. By shifting the burden of localization from physical markers to sophisticated computer vision and deep learning, researchers aim to integrate high-precision guidance into standard clinical workflows without the "hardware tax."

The Problem with Extrinsic Markers

In conventional setups, a navigation system "sees" the patient and tools through artificial proxies. As noted by Ali R. Bani-Hashemi and colleagues, these complex calibration processes and expensive hardware are difficult to integrate into fast-paced operating rooms Interventions under Video-Augmented X-Ray Guidance: Application to Needle Placement.

Physical markers introduce several failure points:

  1. Line-of-Sight Occlusion: External cameras (like those on tripods) are easily blocked by the surgical team Towards markerless surgical tool and hand pose estimation.
  2. Marker Displacement: Adhesive markers can slip on the skin, and rigid frames can be bumped, destroying the spatial registration Multi-View 3D Skin Feature Recognition and Localization for Patient Tracking in Spinal Surgery Applications.
  3. Workflow Disruption: Attaching markers and performing manual "point-to-point" registration adds significant time to a procedure Markerlose Navigation für perkutane Nadelinsertionen.

Tracking Anatomy via Natural Landmarks

The first step in eliminating markers is teaching the system to recognize the patient's own anatomy as a tracking target. Francesca Manni and her team demonstrated that natural skin features, such as moles and pigment spots, can replace adhesive stickers for spinal surgery tracking, achieving sub-millimeter localization Multi-View 3D Skin Feature Recognition and Localization for Patient Tracking in Spinal Surgery Applications.

For internal structures, researchers use geometric invariants. Jacques Feldmar and Nicholas Ayache pioneered the use of "bitangents" and high-dimensional extensions of the Iterative Closest Point (ICP) algorithm to register 3D CT/MRI models to 2D intraoperative images without any markers 3D-2D Projective Registration of Free-Form Curves and Surfaces. More recently, Johannes Bender showed that surgeons can simply draw "visual cues" on a standard monocular endoscope feed, which a differentiable rendering engine then uses to align 3D models to the live view An augmented reality overlay for navigated prostatectomy using fiducial-free 2D-3D registration.

Perceiving Tools Without Modification

Tracking surgical instruments is equally critical. Instead of attaching reflective spheres to a scalpel or drill, new methods learn the "visual signature" of the tools themselves. Austin Reiter and Peter K. Allen used randomized trees to identify naturally occurring landmarks on robotic tools Learning features on robotic surgical tools.

Modern deep learning has accelerated this. Anh Nguyen introduced SimPS-Net, which performs simultaneous segmentation and 3D pose estimation of standard, unmodified tools using a single RGB camera SimPS-Net: Simultaneous Pose & Segmentation Network of Surgical Tools. Similarly, the TAToo framework by Zhaoshuo Li tracks both the patient’s skull and the surgical tool simultaneously using only the stereo microscopic video feed TAToo: Vision-based Joint Tracking of Anatomy and Tool for Skull-base Surgery.

Inside-Out and Integrated Sensing

A second strategy for reducing clutter is "inside-out" tracking: moving the sensors from the room onto the instruments. Benjamin Busam and colleagues applied Visual SLAM (Simultaneous Localization and Mapping) to medical probes. By mounting a miniature camera directly on an ultrasound probe, the system maps the operating room to determine its own position, eliminating the need for an external camera array Markerless Inside-Out Tracking for Interventional Applications.

Augmented Reality (AR) headsets like the HoloLens 2 are also being repurposed as self-contained navigation hubs. Brian A. Hargreaves developed EasyREG, which uses the headset's built-in Time-of-Flight (ToF) depth sensor to perform markerless registration of the patient's body directly in the surgeon's field of view EasyREG: Easy Depth-Based Markerless Registration and Tracking using Augmented Reality Device for Surgical Guidance.

Algorithmic Motion Compensation

Markerless techniques are also solving the problem of patient motion during imaging. In PET and CT scans, even slight movement can blur the results. William Pak Tun and Greg Mori developed a stereo-video system that tracks head pose without markers to correct PET data in real-time Motion estimation for functional medical imaging studies using a stereo video head pose tracking system.

In more advanced applications, researchers are using the imaging modality itself to track motion. Ciprian Catana showed that simultaneous MRI data can be used to "deblur" PET scans MRI-Assisted PET Motion Correction for Neurologic Studies in an Integrated MR-PET Scanner, while Zhuotong Cai developed DL-HMC++, a deep learning model that estimates head motion directly from the raw PET data "cloud" images, removing the need for any external tracking hardware whatsoever PET Head Motion Estimation Using Supervised Deep Learning With Attention.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_markerless_surgical_navigation_and_motio_588cd5ee,
  title = {Markerless Surgical Navigation and Motion Tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/markerless-surgical-navigation-and-motion-tracking/rdr_588cd5ee00fb4292a5aeada720cb82c9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_markerless_surgical_navigation_and_motio_588cd5ee_v1,
  title = {Markerless Surgical Navigation and Motion Tracking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/markerless-surgical-navigation-and-motion-tracking/rdr_588cd5ee00fb4292a5aeada720cb82c9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}