Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning

Zih-Yun Chiu, Florian Richter, Emily K. Funk, Ryan K. Orosco, Michael C. Yip

TL;DR

This paper presents a reinforcement learning (RL) framework for rapid, bimanual suture needle regrasping in robotic surgery. By employing an ego-centric state space and a novel mixed exploration training strategy, the system achieves a 97% success rate in simulation with near-instantaneous planning times (~0.02s), significantly outperforming traditional motion planning algorithms and matching or exceeding human expert speeds in real-world trials.

What Problem Does This Paper Address?

In Robot-Assisted Minimally Invasive Surgery (RAMIS), suturing is one of the most frequent yet time-consuming tasks. A critical sub-step is regrasping: moving the needle from one robotic gripper to the other to achieve the precise orientation required for the next "throw."

Current challenges include:

  1. Complexity: Expert surgeons spend an average of 7.9 seconds per regrasp, and roughly 74% of suture throws require one.
  2. Planning Latency: Traditional sampling-based motion planners (like RRT*) are often too slow for real-time surgery, sometimes taking minutes to find a collision-free path in constrained environments.
  3. Generalization: Most existing automated methods are "hard-coded" for specific robot setups. If the robot's base moves or the workspace changes, the system usually requires tedious recalibration or retraining.

Methods

The authors propose a learning-based approach to generate trajectories for bimanual regrasping rapidly.

1. Ego-Centric State and Action Space

Instead of defining the robot's position relative to a fixed "world" frame, the authors use an ego-centric approach. The state is defined by the relative poses between the two end-effectors and the needle.

  • Why it matters: This makes the learned policy "robot-agnostic." As long as the grippers can reach the needle, the policy doesn't care about the specific joint angles or where the robot's base is located.
Coordinate systems on the needle. (a) shows the needle frame, and (b) shows the grasping point frame.

2. Mixed Exploration Strategy

Training a policy from scratch for such a precise task is difficult. The authors use Deep Deterministic Policy Gradient (DDPG) combined with Behavior Cloning (BC). They introduce a "mixed exploration" strategy:

  • Initially, the agent uses random exploration.
  • As training progresses, "targeted exploration" is introduced, where an expert motion planner (BIT*) provides demonstrations if the RL policy fails.
  • This prevents the agent from getting stuck in "dead zones" while ensuring it doesn't over-rely on potentially sub-optimal expert paths.

3. Reverse Planning for Demonstrations

The space near the needle is like a "funnel"—as the gripper gets closer, the collision-free volume shrinks drastically, making forward planning very difficult. To generate better training data, the authors use reverse planning: they start the planner at the goal (holding the needle) and plan a path away into free space, then reverse that trajectory for the RL agent to learn from.

The "funnel" effect: as the gripper approaches the needle, the ratio of feasible (collision-free) samples drops sharply.

Datasets

The study does not use a pre-existing public dataset. Instead:

  • Simulation Training: Data was generated in a V-REP environment modeled after the da Vinci Research Kit (dVRK).
  • Expert Demos: 9,900 demonstration episodes were generated using the BIT* algorithm in reverse.
  • Evaluation: The policy was tested on 300 randomized simulated trials and then validated on a physical dVRK system.

Key Insights and Ideas

  • Ego-centricity is the "Secret Sauce": By focusing on relative poses rather than absolute coordinates, the authors solved the generalization problem. Figure 3 in the paper illustrates how two completely different robot configurations can share the same "state" in this ego-centric model.
  • The Power of "Mixed" Learning: Pure RL is too slow to converge on this task, and pure Behavior Cloning is too rigid. The gradual transition from random noise to expert-guided exploration (and eventually to the pure RL policy) proved to be the most stable training method.
  • Planning vs. Execution: While traditional planners are "optimal," their high computational cost makes them impractical for fluid surgery. This RL approach moves the heavy computation to the training phase, allowing for millisecond-level inference during the operation.

Key Results

The proposed method, DDPG+BC (mixed), was compared against several state-of-the-art planners.

Metric DDPG+BC (mixed) BIT* (Reverse) RRT* (Forward)
Success Rate (Sim) 97% 72% 18%
Planning Time 0.0212s 2.59s 186.2s
Training curves showing the success rate of different exploration strategies over 500 epochs.

Real-World Performance: * When the needle pose was known exactly, the success rate was 90.5%. * When the needle pose had to be reconstructed from RGB images (a noisier process), the success rate dropped to 73.3%. * Total run time (planning + execution) was roughly 2.8 to 5.1 seconds, significantly faster than the human expert average of 7.9 seconds.

Three successful real-world trials on the da Vinci surgical robot.

Limitations and Future Work

  • Vision is the Bottleneck: The primary cause of failure in the real world was inaccurate needle pose estimation from the camera. If the vision system thinks the needle is 2mm to the left of its actual position, the grasp will fail.
  • Physical Constraints: The current policy does not explicitly account for robot joint limits, which led to some failures in the real-world trials.
  • Future Directions: The authors suggest incorporating Bayesian estimation to improve needle tracking and adding constraints to the RL reward function to avoid joint limits and singular configurations.

Summary updated

Abstract

Regrasping a suture needle is an important yet time-consuming process in suturing. To bring efficiency into regrasping, prior work either designs a task-specific mechanism or guides the gripper toward some specific pick-up point for proper grasping of a needle. Yet, these methods are usually not deployable when the working space is changed. Therefore, in this work, we present rapid trajectory generation for bimanual needle regrasping via reinforcement learning (RL). Demonstrations from a sampling-based motion planning algorithm is incorporated to speed up the learning. In addition, we propose the ego-centric state and action spaces for this bimanual planning problem, where the reference frames are on the end-effectors instead of some fixed frame. Thus, the learned policy can be directly applied to any feasible robot configuration. Our experiments in simulation show that the success rate of a single pass is 97%, and the planning time is 0.0212s on average, which outperforms other widely used motion planning algorithms. For the real-world experiments, the success rate is 73.3% if the needle pose is reconstructed from an RGB image, with a planning time of 0.0846s and a run time of 5.1454s. If the needle pose is known beforehand, the success rate becomes 90.5%, with a planning time of 0.0807s and a run time of 2.8801s.
BibTeX
@inproceedings{chiu2021bimanual,
  author = {Zih-Yun Chiu and Florian Richter and Emily K. Funk and Ryan K. Orosco and Michael C. Yip},
  title = {{Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=QNdq8mJpbGA},
}

Loading related content...

Figure