Single-Shot Pose Estimation of Surgical Robot Instruments' Shafts from Monocular Endoscopic Images

Masakazu Yoshimura, Murilo M. Marinho, Kanako Harada, Mamoru Mitsuishi

Abstract

Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. In this work, we propose a new method to estimate the pose of the surgical instruments' shafts using a monocular endoscope. Our method is based on the use of an automatically annotated training dataset and an improved pose-estimation deep-learning architecture. In preliminary experiments, we show that our method can surpass state of the art vision-based marker-less pose estimation techniques (providing an error decrease of 55% in position estimation, 64% in pitch, and 69% in yaw) by using artificial images.

Introduction

In robot-assisted endonasal surgery—procedures performed through the nose—precision is paramount. To protect critical structures like the pituitary gland, surgical robots utilize "virtual fixtures," which are software-defined safety zones. However, these systems traditionally rely on the robot’s internal kinematic model, which often deviates from reality due to mechanical slack in cable-driven systems, tissue interactions, or thermal changes.

Surgical context showing the proximity of instruments to the pituitary gland and dura mater.

While human surgeons can visually compensate for these discrepancies, autonomous safety systems require real-time "closed-loop" pose estimation. Existing solutions often involve bulky external sensors or expensive stereo cameras that are impractical in the narrow corridors of the nasal cavity. This research proposes a deep-learning-based, "single-shot" method to estimate the 3D pose of instrument shafts using only the standard 2D monocular camera feed already present in the operating room.

Methods

The proposed architecture estimates the 6D pose (position and orientation) of the instrument shaft from a single 2D image. The researchers evolved their approach through several iterations, moving from discrete classification to continuous regression.

Architecture Evolution

Starting with the SSD-6D object detection network, the team identified that treating pose estimation as a classification problem (sorting poses into discrete "bins") lacked the precision required for surgery. They modified the network to perform continuous regression. Their final design, Architecture-D, integrates bounding box and class confidence data back into the pose estimation path. This ensures the network only calculates a pose when it is highly confident an instrument is present in that specific region.

Deep learning architectures evaluated, including the InceptionV4 backbone (a) and the proposed Architecture-D (d) which utilizes feature integration.

Sim-to-Real Bridge

To overcome the lack of annotated surgical data, the authors used a physics-based rendering pipeline in Blender to create a synthetic dataset of 100,000 images. These images featured randomized poses, specular reflections on metallic shafts, and varying lighting conditions. To help the model generalize, 100 real endoscopic images with manual bounding box annotations were mixed into the training set, and uniform random noise was added to simulate the "visual clutter" (blood and tissue) of real surgical environments.

Key Results

The proposed Architecture-D, augmented with noise and real-world data, significantly outperformed baseline adaptations of SSD-6D. In synthetic testing, the method achieved:

  • Position Error Reduction: 55%
  • Pitch Error Reduction: 64%
  • Yaw Error Reduction: 69%

The model achieved an inference speed of 30.4 FPS on a Quadro GV 100, meeting the requirements for real-time surgical feedback. Qualitatively, the model successfully generalized to real endoscopic footage, accurately rendering tool masks that matched manual segmentations.

Conclusion and Future Work

This work demonstrates that continuous regression and synthetic-to-real data augmentation can provide the high-precision pose estimation necessary for robotic safety systems. Future research will focus on incorporating temporal consistency through sequential video data (e.g., LSTMs) to handle momentary occlusions and extending the system to estimate the pose of dexterous instrument tips for task automation.

Summary updated

Abstract

Surgical robots are used to perform minimally invasive surgery and alleviate much of the burden imposed on surgeons. Our group has developed a surgical robot to aid in the removal of tumors at the base of the skull via access through the nostrils. To avoid injuring the patients, a collision-avoidance algorithm that depends on having an accurate model for the poses of the instruments' shafts is used. Given that the model's parameters can change over time owing to interactions between instruments and other disturbances, the online estimation of the poses of the instrument's shaft is essential. In this work, we propose a new method to estimate the pose of the surgical instruments' shafts using a monocular endoscope. Our method is based on the use of an automatically annotated training dataset and an improved pose-estimation deep-learning architecture. In preliminary experiments, we show that our method can surpass state of the art vision-based marker-less pose estimation techniques (providing an error decrease of 55% in position estimation, 64% in pitch, and 69% in yaw) by using artificial images.
BibTeX
@inproceedings{yoshimura2020single,
  author = {Masakazu Yoshimura and Murilo M. Marinho and Kanako Harada and Mamoru Mitsuishi},
  title = {{Single-Shot Pose Estimation of Surgical Robot Instruments' Shafts from Monocular Endoscopic Images}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=ob_D2nUTxb},
}
Figure