Efficiently Calibrating Cable-Driven Surgical Robots With RGBD Fiducial Sensing and Recurrent Neural Networks

Minho Hwang, Brijen Thananjeyan, Samuel Paradis, Daniel Seita, Jeffrey Ichnowski, Danyal Fer, Thomas Low, Ken GoldbergIEEE RA-L 2020 · OpenReview

TL;DR

This paper tackles the challenge of imprecise control in cable-driven surgical robots like the da Vinci Research Kit (dVRK) caused by cable stretching and hysteresis. They introduce a novel, efficient calibration method using RGBD sensing to track fiducial markers and recurrent neural networks (LSTMs) to learn and compensate for these history-dependent errors, significantly improving surgical task performance.

What Problem Does This Paper Address?

Cable-driven surgical robots, while powerful, suffer from inherent inaccuracies. The motors controlling the robot's joints are often far from the actual joint, and the flexible cables connecting them can stretch, exhibit hysteresis (where the output depends on the history of the input), and introduce other complex behaviors. These effects lead to a mismatch between the robot's commanded configuration and its true physical configuration, making precise automated surgical subtasks difficult. Current solutions often rely on manual calibration or human surgeons compensating for these errors in real-time, which is time-consuming and limits automation potential. This paper aims to develop an efficient, automated calibration method to overcome these limitations.

Methods

The core of the proposed method involves:

  • RGBD Fiducial Sensing: They attach simple, 3D-printed spherical fiducials to the robot's arm and end-effector. An RGBD camera (like the Zivid OnePlus) is used to track the precise 3D positions of these fiducials. This provides a ground-truth measurement of the robot's physical configuration, independent of its internal encoders.
  • Kinematic Modeling: Using the tracked fiducial positions, they derive kinematic equations to estimate the robot's true joint angles ($q_p$). This involves a sphere detection algorithm and then using these points to solve for the robot's pose.
  • History-Dependent Modeling: To capture the complex, history-dependent effects of cables (hysteresis, stretch), they employ machine learning models. They investigate 13 different approaches, including linear regression and Recurrent Neural Networks (specifically LSTMs). These models learn to predict the robot's true joint configuration ($q_p$) given its commanded configuration ($q_c$) and a history of past states (forward model), or to predict the commanded configuration given the true configuration and its history (inverse model).
  • Controller Integration: The learned models are integrated into a controller. For the forward model, they use an iterative optimization algorithm that refines commands to reach a target configuration by accounting for predicted errors. For the inverse model, the learned function directly provides the necessary commands.

Datasets

The paper utilizes two primary datasets for training and evaluation:

  • $D_{rand}$ (Random Motions): This dataset consists of randomly sampled robot configurations collected over approximately 66 minutes, containing 4000 data points.
  • $D_{pick}$ (Pick and Place Motions): This dataset is collected during specific pick-and-place motions relevant to surgical tasks, taking about 31 minutes for 1800 data points.

Test datasets are also collected, representing 10% of the training data size. The authors note that training on data distributions similar to the test task (e.g., $D_{pick}$ for the peg transfer task) yields better performance.

Key Insights and Ideas

  • Low-Cost, High-Impact Calibration: The use of inexpensive 3D-printed fiducials and an RGBD camera offers a practical and accessible way to obtain high-fidelity ground-truth measurements, making advanced calibration feasible.
  • Wrist Joints are the Bottleneck: Preliminary analysis reveals that the primary source of error in the dVRK's state estimation comes from the wrist joints (the last three joints), while the arm joints are relatively accurate and decoupled. This allows for focused calibration efforts.
  • History Matters: The paper strongly demonstrates that temporal information (past states) is crucial for accurately modeling and compensating for cable-driven robot dynamics, particularly hysteresis. Simple static models are insufficient.
  • LSTMs Excel at Capturing Dynamics: Recurrent Neural Networks, specifically LSTMs, prove to be highly effective at capturing the history-dependent nature of cable effects, outperforming linear models and feed-forward networks.
  • Task-Specific Training is Key: Training the calibration model on data that closely matches the target task (e.g., pick-and-place motions for a peg transfer task) significantly improves performance compared to training on generic random motions.

Key Results

  • Reduced Tracking Error: The best trained model (RNN-Cmd-Delta forward model) reduced the mean tracking error of the physical robot from 2.96 mm to 0.65 mm on a test set of reference trajectories.
  • Improved Surgical Task Success: In the Fundamentals of Laparoscopic Surgery (FLS) peg transfer task, the proposed calibration method increased the success rate of block transfers from 39.4% (uncalibrated) to 96.7% (using the forward model trained on task-specific data). This performance is comparable to that of an expert surgical resident.
  • Efficient Calibration: Data collection for 1800 samples takes only 31 minutes, and model training takes under 1 minute, making the calibration process highly efficient.
  • Measurement System Accuracy: The fiducial tracking system achieves an RMS error of 0.32 mm in sphere detection and comparable accuracy in joint angle estimation, making it reliable for ground-truth measurement.

Limitations and Future Work

  • Instrument/Arm Specificity: The current method requires collecting a dataset for each specific instrument and arm configuration. The authors suggest future work could involve meta-learning to adapt quickly to new instruments or arms with limited data.
  • Long-Term Dynamics: While the current method addresses immediate cabling effects, the paper acknowledges the need to study how robot dynamics might change over longer periods.
  • Multilateral Manipulation: The authors propose extending the method to calibrate two PSM arms for multilateral surgical manipulation tasks.
  • Other Surgical Tasks: The calibration method is intended to be applied to other surgical tasks like cutting, debridement, and peg transfer with hand-overs.

Summary updated

Abstract

Automation of surgical subtasks using cable-driven robotic surgical assistants (RSAs) such as Intuitive Surgical's da Vinci Research Kit (dVRK) is challenging due to imprecision in control from cable-related effects such as cable stretching and hysteresis. We propose a novel approach to efficiently calibrate such robots by placing a 3D printed fiducial coordinate frames on the arm and end-effector that is tracked using RGBD sensing. To measure the coupling and history-dependent effects between joints, we analyze data from sampled trajectories and consider 13 approaches to modeling. These models include linear regression and LSTM recurrent neural networks, each with varying temporal window length to provide compensatory feedback. With the proposed method, data collection of 1800 samples takes 31 minutes and model training takes under 1 minute. Results on a test set of reference trajectories suggest that the trained model can reduce the mean tracking error of physical robot from 2.96 mm to 0.65 mm. Results on the execution of open-loop trajectories of the FLS peg transfer surgeon training task suggest that the best model increases success rate from 39.4% to 96.7%, producing performance comparable to that of an expert surgical resident. Supplementary materials, including code and 3D-printable models, are available at https://sites.google.com/berkeley.edu/surgical-calibration.
BibTeX
@article{hwang2020efficiently,
  author = {Minho Hwang and Brijen Thananjeyan and Samuel Paradis and Daniel Seita and Jeffrey Ichnowski and Danyal Fer and Thomas Low and Ken Goldberg},
  title = {{Efficiently Calibrating Cable-Driven Surgical Robots With RGBD Fiducial Sensing and Recurrent Neural Networks}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2020},
  url = {https://openreview.net/forum?id=k9PJmqE_XV},
}
Figure