Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning

Zih-Yun Chiu, Florian Richter, Emily K. Funk, Ryan K. Orosco, Michael C. Yip

Abstract

Regrasping a suture needle is an important yet time-consuming process in suturing. To bring efficiency into regrasping, prior work either designs a task-specific mechanism or guides the gripper toward some specific pick-up point for proper grasping of a needle. Yet, these methods are usually not deployable when the working space is changed. Therefore, in this work, we present rapid trajectory generation for bimanual needle regrasping via reinforcement learning (RL). Demonstrations from a sampling-based motion planning algorithm is incorporated to speed up the learning. In addition, we propose the ego-centric state and action spaces for this bimanual planning problem, where the reference frames are on the end-effectors instead of some fixed frame. Thus, the learned policy can be directly applied to any feasible robot configuration. Our experiments in simulation show that the success rate of a single pass is 97%, and the planning time is 0.0212s on average, which outperforms other widely used motion planning algorithms. For the real-world experiments, the success rate is 73.3% if the needle pose is reconstructed from an RGB image, with a planning time of 0.0846s and a run time of 5.1454s. If the needle pose is known beforehand, the success rate becomes 90.5%, with a planning time of 0.0807s and a run time of 2.8801s.

Introduction

In Robot-Assisted Minimally Invasive Surgery (RAMIS), suturing remains a bottleneck for automation. A critical sub-step is needle regrasping—transferring the needle between robotic grippers to achieve the orientation needed for the next "throw." Expert surgeons spend significant time on this (averaging 7.9s per regrasp), and traditional motion planners like RRT* are often too slow for real-time use. Furthermore, existing automated methods often lack generalization, requiring recalibration if the robot's workspace or base position changes.

This paper introduces a reinforcement learning (RL) framework designed for rapid, bimanual needle regrasping. By utilizing an ego-centric state space and a mixed exploration training strategy, the system achieves near-instantaneous planning (~0.02s) and high success rates, matching or exceeding human expert speeds in real-world trials.

Methodology

Ego-Centric State and Action Space

A key innovation of this work is the use of an ego-centric approach. Instead of defining positions relative to a fixed world frame, the state is defined by the relative poses between the two end-effectors and the needle. This makes the policy "robot-agnostic"; as long as the grippers can reach the needle, the specific joint angles or base location do not affect the planning logic.

Coordinate systems used for the ego-centric state space: (a) Needle frame and (b) Grasping point frame.

Mixed Exploration Strategy

Training a policy for high-precision surgical tasks is challenging due to the sparse reward signal. The authors employ Deep Deterministic Policy Gradient (DDPG) combined with Behavior Cloning (BC) and a novel "mixed exploration" strategy. Initially, the agent uses random exploration. As training progresses, "targeted exploration" is introduced, where an expert motion planner (BIT*) provides demonstrations if the RL policy fails. This prevents the agent from getting stuck in local minima while ensuring it doesn't over-rely on potentially sub-optimal expert paths.

Reverse Planning for Demonstrations

To generate high-quality training data, the authors utilize reverse planning. Since the space near the needle is highly constrained (a "funnel" effect), planning forward is difficult. By starting at the goal (holding the needle) and planning a path away into free space, the system generates feasible trajectories that are then reversed for the RL agent to learn from.

Results

Simulation and Real-World Performance

The proposed DDPG+BC (mixed) method was evaluated against standard planners in a V-REP environment, achieving a 97% success rate with a planning time of 0.0212s. This significantly outperformed BIT (72% success, 2.59s) and RRT (18% success, 186.2s).

Validation on a physical da Vinci Research Kit (dVRK) confirmed these results. When the needle pose was known exactly, the success rate reached 90.5%. With needle pose reconstructed from RGB images, the success rate was 73.3%, primarily due to vision-based estimation errors.

Successful real-world bimanual regrasping trials (T1-T3) on the da Vinci surgical robot, showing the state before and after the regrasp.

Total run times (planning plus execution) ranged from 2.8 to 5.1 seconds, which is notably faster than the human expert average of 7.9 seconds.

Conclusion

This work demonstrates that RL, combined with ego-centric representations and expert-guided exploration, can solve complex bimanual manipulation tasks in surgical robotics with high speed and generalization. Future work will focus on improving needle tracking accuracy and incorporating physical constraints like joint limits directly into the RL reward function.

Summary updated

Abstract

Regrasping a suture needle is an important yet time-consuming process in suturing. To bring efficiency into regrasping, prior work either designs a task-specific mechanism or guides the gripper toward some specific pick-up point for proper grasping of a needle. Yet, these methods are usually not deployable when the working space is changed. Therefore, in this work, we present rapid trajectory generation for bimanual needle regrasping via reinforcement learning (RL). Demonstrations from a sampling-based motion planning algorithm is incorporated to speed up the learning. In addition, we propose the ego-centric state and action spaces for this bimanual planning problem, where the reference frames are on the end-effectors instead of some fixed frame. Thus, the learned policy can be directly applied to any feasible robot configuration. Our experiments in simulation show that the success rate of a single pass is 97%, and the planning time is 0.0212s on average, which outperforms other widely used motion planning algorithms. For the real-world experiments, the success rate is 73.3% if the needle pose is reconstructed from an RGB image, with a planning time of 0.0846s and a run time of 5.1454s. If the needle pose is known beforehand, the success rate becomes 90.5%, with a planning time of 0.0807s and a run time of 2.8801s.
BibTeX
@inproceedings{chiu2021bimanual,
  author = {Zih-Yun Chiu and Florian Richter and Emily K. Funk and Ryan K. Orosco and Michael C. Yip},
  title = {{Bimanual Regrasping for Suture Needles using Reinforcement Learning for Rapid Motion Planning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=QNdq8mJpbGA},
}
Figure