Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance

Muhammad Asif Rana, Daphne Chen, Seyed Reza Ahmadzadeh, Jacob Williams, Vivian Chu, Sonia Chernova

Abstract

We contribute a study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these techniques. In particular, we evaluate four approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four manipulation tasks. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.

Introduction

Learning from Demonstration (LfD) is a powerful paradigm that allows non-experts to teach robots new skills through physical guidance, often referred to as kinesthetic teaching. However, the field currently lacks standardized benchmarks, with many algorithms tested only in controlled, "best-case" scenarios. This paper addresses this gap by providing a "reality check" for LfD, evaluating how different algorithms handle the variability of human demonstrators, task constraints, and the need for spatial generalization.

Methods and Experimental Design

The study compares four representative LfD algorithms, each belonging to a distinct mathematical family:

  • CLFDM (Dynamical Systems): Focuses on stable flows toward a goal.
  • ProMP (Probabilistic): Models trajectory distributions to capture timing and variance.
  • TLGC (Geometric): Prioritizes the geometric shape of the motion.
  • TpGMM (Statistical): Uses Gaussian Mixture Models and optimal control for adaptation.
Overview of the benchmarking study design, including participants, tasks, algorithms, and evaluation scale.

To evaluate these methods, the researchers conducted a large-scale study involving nine participants with varying levels of experience (Low, Medium, and High). The resulting 180 models were used to generate 720 robot executions on a Rethink Sawyer robot, which were then rated by 3,600 Amazon Mechanical Turk (AMT) workers on criteria such as efficiency, safety, and success.

Datasets and Tasks

The benchmark includes four manipulation tasks of increasing complexity: Reaching (point goal), Pushing (directional constraint), Pressing (multi-step/segmented), and Writing (strict shape constraint).

The four manipulation tasks (a-d) and their corresponding trajectory data used for training and evaluation.

Key Insights and Results

The study yielded several critical findings that challenge common assumptions in robotics:

The Fallacy of Geometric Metrics

A major finding is that standard error metrics like Mean Squared Error (MSE) are poor predictors of real-world task success. A robot might achieve low MSE but fail a critical task component (like pressing a button), while another might have high MSE but be perceived as successful by human evaluators.

Impact of User Expertise

User experience significantly impacts model performance. There is a notable performance jump between "Low" experience users (no prior robot contact) and "Medium" experience users. Interestingly, the gap between "Medium" and "High" (LfD experts) is smaller, suggesting that basic "robot literacy" is the most critical factor for high-quality data collection.

Task Complexity and Segmentation

For multi-step tasks like "Pressing," standard algorithms often fail unless the motion is manually segmented into "primitives." For instance, segmentation improved the average rating for CLFDM by nearly 2.5 points on a 3-point scale.

Algorithm Strengths

  • Geometric Dominance: TLGC proved most consistent for shape-constrained tasks like Writing and Pushing.
  • Generalization Limits: All algorithms showed a significant drop in performance as the robot's starting position moved further from the training data, highlighting a lack of robust spatial generalization across the board.

Conclusion

This benchmark provides a foundational dataset and clear guidelines for future LfD research. It emphasizes the need for better instructional design for human demonstrators and the importance of moving beyond simple MSE toward human-centric evaluation metrics. Future work will focus on long-horizon tasks and improving the robustness of these algorithms to large spatial perturbations.

Summary updated

Abstract

In this work, we contribute a large-scale study benchmarking the performance of multiple motion-based learning from demonstration approaches. Given the number and diversity of existing methods, it is critical that comprehensive empirical studies be performed comparing the relative strengths of these learning techniques. In particular, we evaluate four different approaches based on properties an end user may desire for real-world tasks. To perform this evaluation, we collected data from nine participants, across four different manipulation tasks with varying starting conditions. The resulting demonstrations were used to train 180 task models and evaluated on 720 task reproductions on a physical robot. Our results detail how i) complexity of the task, ii) the expertise of the human demonstrator, and iii) the starting configuration of the robot affect task performance. The collected dataset of demonstrations, robot executions, and evaluations are being made publicly available. Research insights and guidelines are also provided to guide future research and deployment choices about these approaches.
BibTeX
@inproceedings{rana2020benchmark,
  author = {Muhammad Asif Rana and Daphne Chen and Seyed Reza Ahmadzadeh and Jacob Williams and Vivian Chu and Sonia Chernova},
  title = {{Benchmark for Skill Learning from Demonstration: Impact of User Experience, Task Complexity, and Start Configuration on Performance}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  url = {https://openreview.net/forum?id=if_gl3HEWPq},
}
Figure