Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery

Thomas Probst, Kevis-Kokitsi Maninis, Ajad Chhatkuli, Mouloud Ourak, Emmanuel B. Vander Poorten, Luc Van Gool

Introduction

Retinal microsurgery is a high-stakes environment where surgeons must manipulate membranes thinner than a human hair. While robotic assistants provide the necessary mechanical stability, a critical gap remains: the robot's internal coordinate system must be precisely registered to the surgical field of view seen through a stereo microscope. This registration is notoriously difficult because surgical microscopes have narrow fields of view and shallow depths of field, making standard calibration techniques—such as using checkerboard patterns—unreliable.

Proposed Methodology

The authors introduce a multi-stage pipeline that treats the surgical tool itself as the calibration target, leveraging its known geometry and the robot's kinematic data.

Figure 1: Overview of the proposed pipeline, illustrating the flow from stereo camera input to 3D robot-to-camera registration.

The Pipeline Components:

  1. Tool Landmark Detection: A Stacked Hourglass Network (SHN) is employed to detect three specific keypoints on the surgical tool (the base and two tips). To maintain the precision required for microsurgery, the network is optimized to process full-HD (1080p) images at 35Hz.
  2. Affine Camera Calibration: Due to the long focal length of microscopes relative to the workspace, the authors utilize an affine projection model rather than a standard perspective model. This model is more stable and is solved using a robust Direct Linear Transform (DLT) combined with Bundle Adjustment.
  3. 3D Surface Reconstruction: Semi-dense stereo correspondence is achieved via DeepMatching, and the resulting point cloud is smoothed into a surface using Bicubic B-Splines (BBS).
  4. Registration: The system aligns the camera’s 3D view with the robot’s coordinate system by minimizing the distance between triangulated tool points and the robot's kinematic sensors.

Key Insights: The Importance of Resolution

A significant finding of this research is that image resolution is a primary driver of accuracy. In many deep learning tasks, images are downsampled to save on computation; however, in the "game of microns" that is retinal surgery, this practice significantly degrades performance.

As illustrated in Figure 5, the SHN achieves a 99.7% PCK (Percentage of Correct Keypoints) at a strict threshold when using 1080p images. This high resolution allows the system to achieve accuracy comparable to manual human annotation, which is essential for safe robotic intervention.

Results and Performance

The system demonstrates impressive real-time capabilities and precision:

  • Speed: Tool detection runs at 35Hz, exceeding standard video rates (30Hz) for low-latency feedback.
  • Precision: Controlled tests showed a reconstruction Root Mean Square Error (RMSE) of approximately 25.5 μm.
  • Registration Accuracy: The system recovered tool-to-eye registration with an error below 150 μm after only three frames of movement.

Limitations and Future Outlook

While the results on porcine eye sequences are promising, translating this to in-vivo human surgery presents new challenges. Future iterations must account for the refractive properties of the human cornea and lens, as well as environmental factors like blood, bubbles, and varying tool types. By providing a unified pipeline for detection and registration, this work establishes a foundation for intelligent robotic assistants in the operating room.

Summary updated

BibTeX
@article{probst2018automatic,
  author = {Thomas Probst and Kevis-Kokitsi Maninis and Ajad Chhatkuli and Mouloud Ourak and Emmanuel B. Vander Poorten and Luc Van Gool},
  title = {{Automatic Tool Landmark Detection for Stereo Vision in Robot-Assisted Retinal Surgery}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2018},
  doi = {10.1109/lra.2017.2778020},
}
Figure