Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation

Martin Huber, John Bason Mitchell, Ross Henry, Sébastien Ourselin, Tom Vercauteren, Christos BergelesISMR 2021 · OpenReview

TL;DR

This paper introduces a novel homography-based visual servoing system for robotic endoscopes that enables semi-autonomous manipulation without requiring explicit camera or tool positions, or depth information. By focusing on image-to-image transformations (homographies) and respecting a programmable Remote Center of Motion (RCM), the system allows surgeons to define and navigate between desired views, simplifying complex surgical tasks.

What Problem Does This Paper Address?

Minimally Invasive Surgery (MIS) relies heavily on endoscopic visualization, but manual control can be taxing for surgeons, leading to fatigue and potential errors. Existing robotic endoscope holders and visual servoing methods often depend on knowing precise tool and camera positions, or require depth information, which can be difficult to obtain reliably in dynamic surgical environments. These methods can also be sensitive to calibration errors and patient movement. The paper addresses the need for a more robust and intuitive visual servoing system that can automate camera control by focusing on desired visual scenes rather than precise geometric points, while also adhering to the crucial Remote Center of Motion (RCM) constraint essential for surgical manipulation.

Methods

The core innovation lies in a homography-based visual servoing task. Instead of tracking specific points, the system learns to map the current endoscopic view to a desired view using homographies. A homography is a transformation that relates corresponding points in two images of the same scene taken from different viewpoints. This allows the system to servo towards a desired image appearance rather than a specific geometric configuration.

This homography-based task is integrated with a programmable Remote Center of Motion (RCM). The RCM is a crucial constraint in MIS, ensuring that the instrument enters the body at a fixed point (the trocar) while allowing for manipulation within the body. The paper extends existing RCM control methods by incorporating the homography-based visual servoing objective into a composite Jacobian framework. This allows the robot to simultaneously satisfy the RCM constraint and achieve the desired visual target.

The system uses a graph-based representation of desired views. Surgeons can manually capture and store key views, building a graph where nodes are images and edges represent transitions between them. The system can then automatically plan and execute a path through this graph to reach a selected target view.

Datasets

The paper evaluates its method on an abdominal phantom using a Szabo Pelvic Trainer and a Kyoto Kagaku colon rectum tube to simulate a laparoscopic view. This setup serves as a custom benchmark for evaluating the system's performance in clinically relevant scenarios. No publicly available datasets were used for training or evaluation.

Key Insights and Ideas

  • Scene-centric vs. Point-centric Servoing: The shift from tracking specific points to matching image appearances (via homographies) is a significant departure. This makes the system more robust to variations in lighting, minor occlusions, and the absence of easily identifiable features, as long as the overall scene appearance can be matched.
  • Programmable RCM Flexibility: Using a programmable RCM, as opposed to a fixed mechanical one, allows for greater adaptability and can be integrated with other robotic functionalities.
  • Graph-based View Navigation: Building a graph of desired views provides an intuitive way for surgeons to define targets and for the system to plan complex, multi-step movements. This simplifies the user interaction significantly.
  • Independence from Depth and Explicit Positions: The ability to perform visual servoing without explicit camera or tool positions, or depth information, addresses a major limitation of previous methods and makes the system more practical for real-world surgical applications.

Key Results

The system demonstrated successful visual servoing in three scenarios on the abdominal phantom:

  • Servoing between arbitrary views: The system achieved convergence with a final camera tip deviation of 0.4 mm and an RCM deviation from the trocar of 0.8±0.8 mm.
  • Servoing under tool motion: Despite simulated tool movement, the system converged to desired views with pixel accuracy, showing a final camera position deviation of 1.4 mm.
  • Servoing after phantom repositioning: The system successfully readjusted the view after phantom rotation and tilting, achieving submillimeter camera position convergence and minimizing visual error to pixel range.

The integral term in the controller was found to be crucial for removing steady-state errors in image alignment. The system operates at a frame rate of approximately 30 Hz for image processing and 100 Hz for RCM control.

Limitations and Future Work

  • Initialization of RCM: The RCM still requires initialization, which is a standard procedure but a necessary step.
  • Homography Estimation Noise: The homography estimation can be noisy due to feature sparseness, requiring filtering.
  • Redundant DOF: The robot has one redundant DOF, which means it doesn't always return to its exact initial configuration. The authors suggest this could be addressed by incorporating joint state objectives into the Jacobian's nullspace.
  • Future Work:
    • Incorporating force-torque sensing to update the RCM.
  • Developing more robust homography estimation methods, potentially using deep learning, that are invariant to object motion and feature sparseness.
  • Investigating the use of joint state objectives to manage redundant DOFs.

Summary updated

Abstract

The dominant visual servoing approaches in Minimally Invasive Surgery (MIS) follow single points or adapt the endoscope’s field of view based on the surgical tools’ distance. These methods rely on point positions with respect to the camera frame to infer a control policy. Deviating from the dominant methods, we formulate a robotic controller that allows for image-based visual servoing that requires neither explicit tool and camera positions nor any explicit image depth information. The proposed method relies on homography-based image registration, which changes the automation paradigm from point-centric towards surgical-scene-centric approach. It simultaneously respects a programmable Remote Center of Motion (RCM). Our approach allows a surgeon to build a graph of desired views, from which, once built, views can be manually selected and automatically servoed to irrespective of robot-patient frame transformation changes. We evaluate our method on an abdominal phantom and provide an open source ROS Moveit integration for use with any serial manipulator 3 . A video is provided 4 .
BibTeX
@inproceedings{huber2021homography,
  author = {Martin Huber and John Bason Mitchell and Ross Henry and Sébastien Ourselin and Tom Vercauteren and Christos Bergeles},
  title = {{Homography-based Visual Servoing with Remote Center of Motion for Semi-autonomous Robotic Endoscope Manipulation}},
  booktitle = {ISMR},
  year = {2021},
  url = {https://openreview.net/forum?id=o6GMhRZNcY},
}
Figure