Learning from Extrapolated Corrections

Jason Y. Zhang, Anca D. DraganICRA 2019 · OpenReview

TL;DR

This paper tackles the challenge of teaching robots desired behaviors by learning from user corrections, rather than full demonstrations. They propose framing this learning as an online function approximation problem, where the robot extrapolates a single corrected waypoint to infer the user's full intended trajectory. Their work shows that using non-Euclidean norms for this extrapolation can lead to more accurate robot learning and a better user experience.

What Problem Does This Paper Address?

Teaching robots complex tasks often involves specifying a "cost function" that guides their motion planning. However, manually defining these cost functions is difficult because it's hard to anticipate all scenarios and user preferences. While learning from full demonstrations is an alternative, providing such demonstrations for manipulation tasks can be cumbersome due to the high dimensionality of robot movements. This paper addresses this by focusing on learning from corrections. The idea is that it's much easier for a user to tweak a robot's proposed path at a specific point (e.g., "move this part of the arm here") than to demonstrate the entire desired motion from scratch. The core problem is how the robot can intelligently interpret a single correction to understand the user's overall intent for the entire trajectory.

Methods

The authors frame the problem of interpreting a correction as an online function approximation task. When a user corrects a waypoint $q_t$ to $\tilde{q}_t$, the robot has its current trajectory $\xi$ and the corrected waypoint $\tilde{q}_t$. The goal is to find a new trajectory $\tilde{\xi}$ that passes through $\tilde{q}_t$ at time $t$, while minimally deviating from the original trajectory $\xi$. This deviation is measured using an inner product $A$ in the Hilbert space of trajectories.

The key novelty lies in exploring different choices for this inner product $A$. While a standard Euclidean inner product (leading to $\tilde{\xi} = (\dots, \tilde{q}_t, \dots)$) is simple, the authors hypothesize that non-Euclidean norms can better capture user intent. They explore norms based on:

  • Finite differences: Penalizing changes in velocity, leading to smoother trajectories.
  • Gaussian (RBF) kernels: Allowing for more localized or global propagation of the correction.

The algorithm then uses these extrapolated trajectories to update the robot's internal cost function estimate, effectively learning from the user's feedback.

Datasets

The paper does not introduce new datasets. Instead, it uses simulated environments for evaluation. These environments are procedurally generated with varying numbers of object types (features) and instances of those types, allowing for systematic testing of different complexities. For the user study, real users interact with a JACO2 robot in two distinct simulated environments designed to test different aspects of correction propagation.

Key Insights and Ideas

  • Corrections are a powerful, yet underspecified, signal: Users can easily provide corrections, but a single corrected point doesn't fully define the intended trajectory. The robot must make assumptions to extrapolate.
  • The choice of extrapolation norm matters significantly: Different norms (Euclidean, velocity-based, RBF kernels) lead to different ways the correction propagates through the trajectory. This choice directly impacts how well the robot learns the user's true cost function.
  • Non-Euclidean norms can be superior for user intent: In many scenarios, especially less cluttered ones, norms that consider trajectory smoothness (like velocity-based norms) or have tunable locality (like RBF kernels) allow the robot to better infer what the user intended, leading to more accurate learning.
  • Task-dependent norm selection: The "best" norm isn't universal. In highly cluttered environments, wider propagations can lead to counter-productive updates, suggesting a need for more localized or task-specific norms.

Key Results

  • Simulation: In simpler environments, wider propagations (e.g., larger RBF kernels) generally led to lower costs. However, as environments became more complex, narrower propagations became more effective. The velocity-based norm performed comparably to Euclidean, except in environments with high clutter and many feature types.
  • User Study:
    • Objective: The non-Euclidean (velocities) norm led to significantly lower trajectory costs, supporting the hypothesis that it facilitates better learning.
  • Subjective: Users rated the corrected trajectories as better matching their intentions when using the non-Euclidean norm. They also perceived the robot's learning and their ability to teach it as better with this norm.
  • Location Strategy: Surprisingly, allowing users to choose corrections "anywhere" (rather than at the point of maximum difference) performed better objectively, suggesting users intuitively pick effective correction points.

Limitations and Future Work

  • Norm Selection: The paper acknowledges that the choice of norm is still a commitment. While they show non-Euclidean norms can be better, different tasks might benefit from different norms. Future work could involve learning the desired norm interactively from the user.
  • User Study Realism: The user study presented users with an "optimal" trajectory to recreate, which aids objective evaluation but might not perfectly reflect real-world internal preferences. Future work could explore studies with more naturalistic user goals.
  • Intermediate Trajectory Inference: The current method infers an intended trajectory before updating the cost function. Future work could explore methods that directly learn from corrections without this intermediate step, potentially by reasoning about action values or using more advanced approximation techniques.
  • Clutter and Complexity: The performance of different norms varied with environmental complexity and clutter, highlighting the need for adaptive or context-aware norm selection.

Summary updated

Abstract

Our goal is to enable robots to learn cost functions from user guidance. Often it is difficult or impossible for users to provide full demonstrations, so corrections have emerged as an easier guidance channel. However, when robots learn cost functions from corrections rather than demonstrations, they have to extrapolate a small amount of information - the change of a waypoint along the way - to the rest of the trajectory. We cast this extrapolation problem as online function approximation, which exposes different ways in which the robot can interpret what trajectory the person intended, depending on the function space used for the approximation. Our simulation results and user study suggest that using function spaces with non-Euclidean norms can better capture what users intend, particularly if environments are uncluttered. This, in turn, can lead to the robot learning a more accurate cost function and improves the user's subjective perceptions of the robot.
BibTeX
@inproceedings{zhang2019learning,
  author = {Jason Y. Zhang and Anca D. Dragan},
  title = {{Learning from Extrapolated Corrections}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=OyyblApZRX},
}
Figure