Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

Muhammad Asif Rana, Daphne Chen, Jacob Williams, Vivian Chu, Seyed Reza Ahmadzadeh, Sonia Chernova

This paper provides a much-needed "reality check" for the field of Learning from Demonstration (LfD). While researchers have developed dozens of algorithms to help robots learn from humans, we rarely see them compared head-to-head under the messy conditions of the real world.

The authors present a large-scale benchmark evaluating four major LfD approaches across varying task complexities, demonstrator experience levels, and generalization scenarios. Their findings challenge some common assumptions in robotics—most notably, that our standard mathematical metrics for "accuracy" actually correlate with task success.

TL;DR

This study benchmarks four popular motion-based LfD algorithms using 720 robot executions and 3,600 human evaluations. It reveals that while geometric approaches excel at shape-constrained tasks, no single algorithm is a "silver bullet," and standard error metrics like Mean Squared Error (MSE) are surprisingly poor predictors of real-world performance.

What Problem Does This Paper Address?

Learning from Demonstration (LfD) allows non-experts to teach robots new skills by physically guiding them through a motion (kinesthetic teaching). However, the field is fragmented. Most papers introduce a new algorithm and test it in a controlled "best-case" scenario.

There is a significant gap in understanding how these algorithms perform when:

  1. The person giving the demonstration isn't a robotics expert.
  2. The task has different constraints (e.g., "touch this point" vs. "follow this specific S-curve").
  3. The robot has to start the task from a position it never saw during training.

Methods

The researchers compared four representative algorithms, each from a different mathematical "family":

  • CLFDM (Dynamical Systems): Learns a stable "flow" toward a goal; great for reaching but can be unpredictable in its path.
  • ProMP (Probabilistic): Learns a distribution of trajectories; good for capturing timing and variance.
  • TLGC (Geometric): Fits a "generalized cylinder" around demonstrations; explicitly prioritizes the shape of the motion.
  • TpGMM (Statistical): Uses Gaussian Mixture Models and optimal control (LQR) to adapt to changing object locations.

They tested these using a Rethink Sawyer robot and evaluated the results using 3,600 ratings from Amazon Mechanical Turk (AMT) workers, who judged the robot's efficiency, safety, and success.

Datasets

The authors created a new, publicly available benchmark dataset:

  • Participants: 9 humans categorized by experience (Low, Medium, and High).
  • Tasks: Four manipulation tasks with increasing constraints: Reaching (point goal), Pushing (directional constraint), Pressing (multi-step/segmented), and Writing (strict shape constraint).
  • Scale: 180 trained models resulting in 720 unique robot execution videos.
  • Validation: They also used the standard LASA handwriting dataset for initial parameter tuning to ensure a fair comparison.

Key Insights and Ideas

  • The "MSE Lie": Perhaps the most striking finding is that Mean Squared Error (MSE)—the industry standard for measuring how well a robot "learned" a path—does not correlate with human ratings of success. A robot can have a very low MSE but fail to actually press a button, or a high MSE but still complete a task perfectly in the eyes of a user.
  • Experience is a Threshold, Not a Linear Scale: There was a significant performance jump between "Low" experience users (no robot contact) and "Medium" experience users (general robotics knowledge). However, the gap between "Medium" and "High" (LfD experts) was much smaller, suggesting that basic "robot literacy" is the primary hurdle for quality data collection.
  • Segmentation is Essential: For complex tasks like "Pressing" (which involves two distinct movements), algorithms like CLFDM failed miserably unless the motion was manually broken into "primitives." Segmentation turned a failing model into a highly successful one.
  • Geometric over Probabilistic for Shapes: For tasks where the path matters (like Writing), the geometric approach (TLGC) was the only one that generalized well. Probabilistic and statistical methods tended to "smooth out" the shape too much, resulting in illegible motions.

Key Results

  • Generalization Decay: Performance for all algorithms dropped significantly as the robot's starting position moved further from the training data. No algorithm was truly "robust" to large spatial perturbations.
  • TLGC Dominance in Constraints: TLGC was the most consistent across Pushing, Writing, and Pressing tasks.
  • User Expertise: Models trained on "High" experience demonstrations received statistically significant higher ratings ($p < 0.05$) than those from "Low" experience users.
  • The Power of Primitives: In the Pressing task, using motion segmentation improved the average rating for CLFDM by nearly 2.5 points (on a 0-3 scale).

Limitations and Future Work

The authors acknowledge that they used a "one-size-fits-all" parameter set for the algorithms to simulate a real-world deployment where a novice wouldn't know how to tune hyperparameters. While fair, this might underrepresent the peak potential of some algorithms.

Future work should investigate instructional design: how can we provide better "training for the trainers" so that even "Low" experience users provide high-quality demonstrations? Additionally, the field needs to move toward more complex, long-horizon tasks that go beyond simple reaching and pushing.

Summary updated

Abstract

We contribute a study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these techniques. In particular, we evaluate four approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four manipulation tasks. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.
BibTeX
@inproceedings{rana2020benchmark,
  author = {Muhammad Asif Rana and Daphne Chen and Jacob Williams and Vivian Chu and Seyed Reza Ahmadzadeh and Sonia Chernova},
  title = {{Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=if_gl3HEWPq},
}

Loading related content...

Figure