SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics

Yang Li, Florian Richter, Jingpei Lu, Emily K. Funk, Ryan K. Orosco, Jianke Zhu, Michael C. Yip

TL;DR

The SuPer (Surgical Perception) framework enables robotic surgeons to "see" and interact with soft tissue by combining model-based tracking for rigid tools with model-free tracking for deforming environments. By integrating these two streams into a closed-loop controller, the system successfully performed autonomous tissue manipulation tasks on a da Vinci Surgical System with high precision.

What Problem Does This Paper Address?

While robotic systems like the da Vinci have revolutionized surgery through teleoperation, true autonomy remains a "holy grail." The primary obstacle is perception: surgical environments are unstructured, poorly lit, and filled with soft tissues that deform unpredictably when touched.

Most traditional robotic control algorithms rely on "structured" environments where objects don't change shape. In surgery, if a robot moves to a coordinate to grasp a piece of liver, the act of moving or breathing might shift that tissue by centimeters. To achieve autonomy, a robot needs a framework that can simultaneously track its own tools (which have known mechanical properties) and the "squishy" environment (which does not) in real-time.

Methods

The authors propose a dual-track perception framework that separates the "known" from the "unknown":

  1. Surgical Tool Tracking (Model-Based): Since we know the robot's geometry and kinematics, the authors use a particle filter to localize the tool. They fuse the robot's internal encoder data with visual features (manually painted markers and tool edges) to estimate the tool's exact pose in the camera frame.
  2. Deformable Tissue Tracking (Model-Free): To map the tissue, the system uses stereo-endoscopic images to create a 3D point cloud of "surfels" (surface elements). Because tracking every single point is computationally expensive, they use an Embedded Deformable (ED) graph—a sparse skeleton that drives the deformation of the entire surface model.
  3. The "Masking" Trick: To prevent the tissue tracker from getting confused by the robot arm moving in front of it, the framework generates a virtual "mask" of the tool (based on the tool tracker) and subtracts it from the visual data before the tissue model is updated.
  4. Closed-Loop Control: These two streams are fed into a controller on the da Vinci Research Kit (dVRK). The robot doesn't just move to a fixed coordinate; it moves to a coordinate that is constantly being updated by the perception framework as the tissue stretches and shifts.

Datasets

  • SuPer Dataset: The authors introduced and released a new dataset featuring tissue manipulation sequences with the da Vinci Surgical System to encourage further research.
  • Validation Data: They utilized existing 3D reconstruction datasets (specifically from Innmann et al., 2016) to test the robustness of their deformable tracking.

Key Insights and Ideas

  • Hybrid Tracking is Essential: You cannot treat a surgical scene as a single vision problem. By using a model-based approach for the robot and a model-free approach for the tissue, the system leverages the strengths of both: the precision of kinematics and the flexibility of computer vision.
  • Perception-in-the-Loop: Most prior work focused on either tracking or control. SuPer demonstrates that perception must be an active part of the control loop to handle the "dynamic" nature of surgery.
  • Efficiency through Sparsity: Using an ED graph allows the system to capture complex, non-rigid deformations without needing the massive computational power required for dense, per-pixel tracking.

Key Results

  • Successful Manipulation: In experiments using chicken skin, the robot successfully grasped and tugged the same point on the tissue five times in a row, even as the tissue deformed significantly.
  • Robustness: Without the deformable tracking (using only a static map), the system failed or the computer crashed due to memory overflow after just three grasps.
  • Accuracy: The tool tracker achieved a Mean IoU (Intersection over Union) of 82.4% at 30 frames per second.
  • Stability: The SuPer framework showed significantly lower reprojection errors and higher stability compared to standard "off-the-shelf" feature tracking methods like SURF, which often lose track during large deformations.

Limitations and Future Work

  • Sensor Noise: The accuracy is currently limited by the noise inherent in stereo-endoscopic depth maps. High-quality depth sensing remains a hardware bottleneck.
  • Manual Markers: The current tool tracker relies on painted markers. The authors suggest that moving toward markerless detection (using deep learning tools like DeepLabCut) is a necessary next step for clinical viability.
  • Topological Changes: The current framework handles stretching and shifting but does not yet account for "topological changes"—the fancy way of saying cutting or suturing, where the tissue is actually divided or joined.
  • Clinical Realism: Future iterations will need to handle the visual "mess" of a real surgery, including blood, smoke from cautery, and varying lighting conditions.

Summary updated

Abstract

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this letter, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.
BibTeX
@article{li2020super,
  author = {Yang Li and Florian Richter and Jingpei Lu and Emily K. Funk and Ryan K. Orosco and Jianke Zhu and Michael C. Yip},
  title = {{SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=IjAWRvxRS_},
}

Loading related content...

Figure