SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics

Yang Li, Florian Richter, Jingpei Lu, Emily K. Funk, Ryan K. Orosco, Jianke Zhu, Michael C. Yip

Abstract

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.

Introduction

While robotic systems like the da Vinci have revolutionized surgery through teleoperation, true autonomy remains a significant challenge. The primary obstacle is perception: surgical environments are unstructured, poorly lit, and filled with soft tissues that deform unpredictably when manipulated. Most traditional robotic control algorithms rely on "structured" environments where objects maintain their shape. In surgery, physiological movements like breathing or the act of grasping tissue can shift targets by centimeters. To achieve autonomy, a robot requires a framework capable of simultaneously tracking its own rigid tools and the highly deformable environment in real-time.

Methodology

The SuPer (Surgical Perception) framework addresses these challenges by employing a dual-track perception system that separates known robotic components from unknown biological structures.

The SuPer framework pipeline, integrating tool tracking, tissue reconstruction, and closed-loop robotic control.

Dual-Track Perception

  1. Surgical Tool Tracking (Model-Based): Leveraging known robot geometry and kinematics, the framework uses a particle filter to localize the tool. It fuses internal encoder data with visual features—such as manually painted markers and tool edges—to estimate the tool's precise pose within the camera frame.
  2. Deformable Tissue Tracking (Model-Free): The system processes stereo-endoscopic images to generate a 3D point cloud of "surfels" (surface elements). To maintain computational efficiency, an Embedded Deformable (ED) graph acts as a sparse skeleton, driving the deformation of the entire surface model.

Integration and Control

A critical component of this framework is the "masking" technique. To prevent the tissue tracker from being confounded by the robot arm, the system generates a virtual mask of the tool based on the tool tracker's output. This mask is subtracted from the visual data before updating the tissue model. These synchronized data streams are fed into a closed-loop controller on the da Vinci Research Kit (dVRK), allowing the robot to update its target coordinates dynamically as the tissue stretches and shifts.

Experimental Results

The framework was validated through autonomous tissue manipulation tasks using chicken skin. The system demonstrated high precision, successfully grasping and tugging the same tissue point five times consecutively, even under significant deformation.

Demonstration of the SuPer framework performing autonomous tissue manipulation, showing the raw endoscopic view (top-left), the reconstructed 3D model (top-right), the physical manipulation (bottom-left), and the real-time digital twin (bottom-right).

Key performance metrics include:

  • Accuracy: The tool tracker achieved a Mean Intersection over Union (IoU) of 82.4% at 30 frames per second.
  • Robustness: Without deformable tracking (using a static map), the system failed or suffered memory overflows after just three grasps.
  • Stability: SuPer exhibited significantly lower reprojection errors compared to standard feature tracking methods like SURF, which frequently lose track during large deformations.

Discussion and Future Work

While the SuPer framework represents a significant step toward surgical autonomy, several limitations remain. Current accuracy is constrained by the noise inherent in stereo-endoscopic depth maps, highlighting a need for improved hardware. Additionally, the reliance on manual markers for tool tracking suggests a transition to markerless deep learning-based detection is necessary for clinical application.

Future research will focus on handling "topological changes," such as cutting or suturing, where tissue is divided or joined. Furthermore, the framework must be adapted to handle the complex visual environment of live surgery, including the presence of blood, cautery smoke, and variable lighting. To support these efforts, the authors have released the SuPer Dataset, featuring tissue manipulation sequences for the research community.

Summary updated

Abstract

Traditional control and task automation have been successfully demonstrated in a variety of structured, controlled environments through the use of highly specialized modeled robotic systems in conjunction with multiple sensors. However, the application of autonomy in endoscopic surgery is very challenging, particularly in soft tissue work, due to the lack of high-quality images and the unpredictable, constantly deforming environment. In this work, we propose a novel surgical perception framework, SuPer, for surgical robotic control. This framework continuously collects 3D geometric information that allows for mapping a deformable surgical field while tracking rigid instruments within the field. To achieve this, a model-based tracker is employed to localize the surgical tool with a kinematic prior in conjunction with a model-free tracker to reconstruct the deformable environment and provide an estimated point cloud as a mapping of the environment. The proposed framework was implemented on the da Vinci Surgical System in real-time with an end-effector controller where the target configurations are set and regulated through the framework. Our proposed framework successfully completed soft tissue manipulation tasks with high accuracy. The demonstration of this novel framework is promising for the future of surgical autonomy. In addition, we provide our dataset for further surgical research.
BibTeX
@article{li2020super,
  author = {Yang Li and Florian Richter and Jingpei Lu and Emily K. Funk and Ryan K. Orosco and Jianke Zhu and Michael C. Yip},
  title = {{SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=IjAWRvxRS_},
}
Figure