Toward Learning Context-Dependent Tasks from Demonstration for Tendon-Driven Surgical Robots

Yixuan Huang, Michael Bentley, Tucker Hermans, Alan KuntzarXiv 2021 · arXiv

TL;DR

Tendon-driven continuum robots hold great promise for minimally invasive surgery but are notoriously difficult to automate due to complex kinematics and highly variable patient anatomy. This paper introduces a Learning from Demonstration (LfD) framework that trains a neural network to map environmental context directly to workspace trajectories. This allows the robot to autonomously adapt and execute surgical tasks in novel environments after watching only a few dozen human demonstrations.

What Problem Does This Paper Address?

Continuum robots can bend and snake through complex pathways, making them ideal for navigating tight anatomical spaces like the pleural cavity around the lungs. However, automating these robots presents two major hurdles:

  1. Complex Kinematics: Their flexible backbones are actuated by non-linear tendon routing, making traditional kinematic modeling and motion planning mathematically grueling.
  2. Context Dependence: No two surgical environments are identical. A robot must alter its trajectory based on context—such as the size, shape, and location of a patient's specific anatomy.

Hard-coding rules for every possible anatomical variation is impossible. This paper tackles the challenge of teaching a continuum robot to infer both the task goals and environmental constraints directly from human expert demonstrations, generalizing the learned behavior to unseen scenarios.

Methods

The authors frame the challenge as a context learning problem and evaluate three different mapping models.

An essential design choice was decoupling task learning from robot kinematics. Instead of forcing the machine learning models to predict complex joint/tendon configurations, the models are trained to output a workspace trajectory—a sequence of 3D waypoints for the robot's tip. A standard iterative Inverse Kinematics (IK) solver then translates these 3D waypoints into actual robot motor commands.

The three evaluated models map a context vector $\kappa$ (containing geometric features like position and scale) to the workspace trajectory:

  1. Linear Ridge Regression: A simple baseline mapping context linearly to trajectories.
  2. Kernel Ridge Regression: A non-linear approach utilizing Radial Basis Function (RBF) kernels to capture more intricate environmental relationships.
  3. Neural Trajectory Network: A feed-forward neural network with two hidden layers (128 neurons each) and ReLU activations, trained via Mean Squared Error (MSE) loss to output the entire trajectory sequence simultaneously.

Datasets

Because no standard benchmarks exist for context-dependent continuum robot tasks, the authors collected three distinct datasets using a haptic teleoperation setup in simulation:

  • Plane Task (50 demonstrations): Tracing a self-intersecting "eight" shape on a flat surface where the curve's starting position, width, and height vary.
  • Double Sphere Task (20 training / 10 test cases): Tracing a continuous path across the surfaces of two adjacent spheres with randomly varied radii and positions.
  • Anatomical Task (20 training / 10 test cases): Tracing a diamond-shaped path inside a simulated human pleural space (the cavity between the chest wall and lungs), segmented from a real patient's CT scan. The anatomy's position and scale were randomly varied to simulate different patient sizes and entry points.

Key Insights and Ideas

  • Workspace Trajectory Prediction is Key: By training the models on tip positions rather than joint angles, the system avoids having to learn the highly non-linear physics of the robot itself, drastically simplifying the learning architecture.
  • Implicit Constraint Learning: The models don't just learn where to go; they implicitly learn environmental constraints. For instance, in the sphere and anatomical tasks, the models successfully learned to keep the robot tip sliding along a curved surface purely by mimicking human behavior.
  • High Data Efficiency: While deep learning often demands thousands of data points, the Neural Trajectory Network achieved near-human performance with as few as 20 to 50 demonstrations.

Key Results

The performance of the trajectories was evaluated using the discrete Fréchet distance, which measures the similarity between the generated curves and human reference curves.

  • Non-Linear Dominance: In all tasks, the non-linear models (Kernel Regression and the Neural Network) significantly outperformed the Linear Ridge Regression baseline.
  • Neural Network Superiority: The Neural Trajectory Network consistently achieved the lowest error and lowest variance across the tasks.
  • Human-Level Performance: On the Plane Task, when trained on 50 demonstrations, the neural network's trajectory variance matched the natural variance found between different human expert runs, effectively achieving human-level execution.
Task Linear Regression Error (m) RBF Kernel Error (m) Neural Network Error (m)
Double Sphere ~0.0170 ~0.0055 ~0.0015
Anatomical ~0.0090 ~0.0085 ~0.0035

(Note: Values approximated from Fig. 5 data plots)

Limitations and Future Work

  • Manual Context Input: Currently, the context vector $\kappa$ (such as the exact scale factor or sphere radius) must be explicitly provided to the model. In a real surgical setting, this context needs to be extracted automatically via computer vision (e.g., endoscopic cameras) or intraoperative medical imaging.
  • Simulation Only: The framework has only been validated in simulated environments. Physical realities like tendon friction, hysteresis, and tissue deformation will introduce new challenges on physical hardware.
  • Scope of Robots: The method was only tested on a single type of tendon-driven robot. The authors intend to test the generalizability of this framework on other continuum architectures, such as concentric tube robots.

Summary updated

Abstract

Tendon-driven robots, a type of continuum robot, have the potential to reduce the invasiveness of surgery by enabling access to difficult-to-reach anatomical targets. In the future, the automation of surgical tasks for these robots may help reduce surgeon strain in the face of a rapidly growing population. However, directly encoding surgical tasks and their associated context for these robots is infeasible. In this work we take steps toward a system that is able to learn to successfully perform context-dependent surgical tasks by learning directly from a set of expert demonstrations. We present three models trained on the demonstrations conditioned on a vector encoding the context of the demonstration. We then use these models to plan and execute motions for the tendon-driven robot similar to the demonstrations for novel context not seen in the training set. We demonstrate the efficacy of our method on three surgery-inspired tasks.
BibTeX
@misc{huang2021toward,
  author = {Yixuan Huang and Michael Bentley and Tucker Hermans and Alan Kuntz},
  title = {{Toward Learning Context-Dependent Tasks from Demonstration for Tendon-Driven Surgical Robots}},
  year = {2021},
  eprint = {2110.07789},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.07789},
}
Figure