A surgical system for automatic registration, stiffness mapping and dynamic image overlay

Nicolas Zevallos, Rangaprasad Arun Srivatsan, Hadi Salman, Lu Li, Jianing Qian, Saumya Saxena, Mengyun Xu, Kartik Patath, Howie Choset

Abstract

In this paper we develop a surgical system using the da Vinci research kit (dVRK) that is capable of autonomously searching for tumors and dynamically displaying the tumor location using augmented reality. Such a system has the potential to quickly reveal the location and shape of tumors and visually overlay that information to reduce the cognitive overload of the surgeon. We believe that our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and the shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on phantom prostate models with embedded stiff inclusions.

Introduction

Robot-Assisted Minimally Invasive Surgery (RMIS) provides surgeons with enhanced dexterity and visualization. However, a significant drawback is the complete lack of haptic feedback. Surgeons must rely on "mental mapping"—aligning pre-operative scans (CT/MRI) with the live surgical field—to locate tumors hidden beneath tissue surfaces. This process is cognitively demanding and prone to error. This research introduces a unified framework for the da Vinci Research Kit (dVRK) that automates tumor localization and provides a dynamic 3D stiffness "heat-map" overlay to assist surgeons.

Methodology and Experimental Setup

The proposed system integrates four primary modules into a cohesive pipeline. First, Registration aligns the pre-operative 3D model with the live camera view using a Bingham distribution-based filtering approach. This probabilistic method offers greater robustness to noise than standard algorithms and achieves registration in approximately 2 seconds.

Second, the system performs Autonomous Palpation using Active Level Set Estimation (LSE) driven by Gaussian Processes. This active learning strategy allows the robot to intelligently select probing points to efficiently define the boundary between soft healthy tissue and stiff tumorous tissue, avoiding redundant measurements.

Third, Force Sensing is achieved via a custom miniature tri-axial sensor mounted on a dVRK needle driver, utilizing Force-Sensitive Resistors (FSRs) to measure contact forces. Finally, a Dynamic AR Overlay is generated using the Visualization Toolkit (VTK), texturing the stiffness map onto the registered 3D model in real-time.

The experimental setup featuring the da Vinci Research Kit (dVRK), stereo camera, and prostate phantom with embedded stiff inclusions.

The system was validated using custom prostate phantoms fabricated from Ecoflex 00-10 silicone to mimic human tissue. Stiff plastic inclusions were embedded at varying depths to simulate tumors. Pre-operative geometric models of these phantoms served as the ground truth for registration and mapping tasks.

Results and Discussion

The Bingham-based registration achieved a mean Root Mean Square Error (RMSE) of 1.5mm. Camera-to-robot calibration yielded an RMSE of 2.37mm. Qualitatively, the system successfully identified the shape and location of embedded inclusions. After only 10 probings, the AR overlay provided a clear visual representation of the hidden tumor, closely matching the ground truth.

Visual progression of the system's output: from original stereo images to online registration and the final dynamic stiffness mapping overlay.

This framework closes the loop on surgical AR by providing a dynamic overlay that updates based on real-time physical interaction rather than relying solely on static pre-operative plans. While effective on silicone phantoms, future work will address non-rigid registration to handle organ deformation and tool segmentation to prevent instruments from obscuring the AR overlay. Testing on ex-vivo biological tissues is also planned to further validate the system's clinical potential.

Summary updated

Abstract

In this paper we develop a surgical system using the da Vinci research kit (dVRK) that is capable of autonomously searching for tumors and dynamically displaying the tumor location using augmented reality. Such a system has the potential to quickly reveal the location and shape of tumors and visually overlay that information to reduce the cognitive overload of the surgeon. We believe that our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and the shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on phantom prostate models with embedded stiff inclusions.
BibTeX
@inproceedings{zevallos2018surgical,
  author = {Nicolas Zevallos and Rangaprasad Arun Srivatsan and Hadi Salman and Lu Li and Jianing Qian and Saumya Saxena and Mengyun Xu and Kartik Patath and Howie Choset},
  title = {{A surgical system for automatic registration, stiffness mapping and dynamic image overlay}},
  booktitle = {ISMR},
  year = {2018},
  url = {https://openreview.net/forum?id=pw_aiCGxZIR},
}
Figure