A surgical system for automatic registration, stiffness mapping and dynamic image overlay

Nicolas Zevallos, Rangaprasad Arun Srivatsan, Hadi Salman, Lu Li, Jianing Qian, Saumya Saxena, Mengyun Xu, Kartik Patath, Howie Choset

This paper, presented by researchers from Carnegie Mellon University’s Robotics Institute, introduces a unified surgical framework for the da Vinci Research Kit (dVRK). The system autonomously locates tumors through robotic palpation and overlays a dynamic, 3D stiffness "heat-map" onto the surgeon’s view using augmented reality (AR).

TL;DR

The researchers developed a system that allows surgical robots to autonomously "feel" for tumors and visually project their hidden locations directly onto the surgeon's display. By combining probabilistic registration, active learning for tumor search, and custom force sensing, the system reduces the cognitive load on surgeons who currently lack haptic feedback during robotic procedures.

What Problem Does This Paper Address?

In Robot-Assisted Minimally Invasive Surgery (RMIS), surgeons benefit from high dexterity but suffer from a total lack of haptic (touch) feedback. To find a tumor hidden beneath the tissue surface, a surgeon must mentally align pre-operative scans (like CT or MRI) with the live 2D/3D camera feed. This "mental mapping" is cognitively exhausting and prone to error.

While previous research has looked at force sensing or AR overlays in isolation, there hasn't been a unified system that handles the entire pipeline: registering the organ model to the camera, autonomously searching for the tumor, and updating a visual overlay in real-time as new data comes in.

Methods

The authors propose a modular framework consisting of four main components:

  1. Registration: To align the pre-operative 3D model with the live camera view, they use a Bingham distribution-based filtering approach. This probabilistic method is more robust to noise than the standard Iterative Closest Point (ICP) algorithm and can update the organ's pose in about 2 seconds.
  2. Autonomous Palpation: Instead of a slow, exhaustive grid search, the system uses Active Level Set Estimation (LSE) driven by Gaussian Processes (GP). This "active learning" approach allows the robot to intelligently choose the next point to poke (palpate) to most efficiently define the boundary between healthy (soft) and tumorous (stiff) tissue.
  3. Force Sensing: The team utilized a custom-developed, miniature tri-axial force sensor mounted on a dVRK needle driver. This sensor uses Force-Sensitive Resistors (FSRs) to measure contact forces along the surface normal.
  4. Dynamic AR Overlay: Using the Visualization Toolkit (VTK), the system generates a stiffness map that is textured onto the registered 3D model. As the robot probes the tissue, the "heat-map" updates in the surgeon's stereo viewer, revealing the tumor's shape and location.

Datasets

The study did not use standard public datasets, as the research involves physical robotic interaction. Instead, the system was evaluated using:

  • Custom Prostate Phantoms: Fabricated from Ecoflex 00-10 silicone to mimic human tissue.
  • Stiff Inclusions: Plastic discs were embedded within the phantoms at varying depths to simulate tumors.
  • Pre-operative Models: A priori geometric models of the phantoms were used for the registration task.

Key Insights and Ideas

  • Closing the Loop on AR: Most AR in surgery is "static"—it shows a pre-operative plan that might be misaligned. This system is "dynamic," meaning the overlay changes based on what the robot is actually "feeling" in real-time.
  • Efficiency through Active Learning: By using LSE, the robot doesn't waste time probing areas it already knows are soft. It focuses its efforts on the "ambiguous" regions near the suspected tumor edges.
  • Robot-Agnostic Framework: While demonstrated on the dVRK, the authors emphasize that their software architecture is modular and could theoretically be applied to other robotic surgical platforms.

Key Results

  • Registration Accuracy: The Bingham-based registration achieved a mean Root Mean Square Error (RMSE) of 1.5mm with a computation time of approximately 2 seconds.
  • Camera-to-Robot Calibration: Using a 6-point calibration method, they achieved an RMSE of 2.37mm.
  • Tumor Localization: The system successfully identified the location and shape of embedded inclusions. Qualitative results show that after just 10 probings, the AR overlay provided a clear visual representation of the hidden "tumor" that matched the ground truth model.

Limitations and Future Work

  • Deformation Handling: The current system assumes the organ does not undergo large global deformations during the procedure. In real surgery, organs shift and deform significantly; the authors plan to incorporate non-rigid registration in the future.
  • Obstruction: Currently, the overlaid stiffness map can be obscured by the surgical tools themselves. Future iterations will look into tool segmentation to ensure the overlay appears "behind" the instruments.
  • Clinical Realism: The experiments were limited to silicone phantoms. The next steps involve testing on ex-vivo (animal) organs to validate the force sensing and mapping in more complex, biological environments.

Summary updated

Abstract

In this paper we develop a surgical system using the da Vinci research kit (dVRK) that is capable of autonomously searching for tumors and dynamically displaying the tumor location using augmented reality. Such a system has the potential to quickly reveal the location and shape of tumors and visually overlay that information to reduce the cognitive overload of the surgeon. To the best of our knowledge, our approach is one of the first to incorporate state-of-the-art methods in registration, force sensing and tumor localization into a unified surgical system. First, the preoperative model is registered to the intra-operative scene using a Bingham distribution-based filtering approach. An active level set estimation is then used to find the location and the shape of the tumors. We use a recently developed miniature force sensor to perform the palpation. The estimated stiffness map is then dynamically overlaid onto the registered preoperative model of the organ. We demonstrate the efficacy of our system by performing experiments on phantom prostate models with embedded stiff inclusions.
BibTeX
@inproceedings{zevallos2018surgical,
  author = {Nicolas Zevallos and Rangaprasad Arun Srivatsan and Hadi Salman and Lu Li and Jianing Qian and Saumya Saxena and Mengyun Xu and Kartik Patath and Howie Choset},
  title = {{A surgical system for automatic registration, stiffness mapping and dynamic image overlay}},
  booktitle = {ISMR},
  year = {2018},
  url = {https://openreview.net/forum?id=pw_aiCGxZIR},
}

Loading related content...

Figure