Physical interaction as communication: Learning robot objectives online from human corrections

Dylan P. Losey, Andrea Bajcsy, Marcia K. O'Malley, Anca D. Dragan

TL;DR

This paper proposes a novel approach where robots learn human preferences online by interpreting physical interactions as corrections, rather than mere disturbances. This allows robots to adapt their behavior mid-task, leading to improved performance and human satisfaction.

What Problem Does This Paper Address?

Robots often work in close proximity to humans, leading to inevitable physical interactions. Traditionally, these interactions are treated as disturbances that the robot should reject or avoid. However, this paper argues that these interactions are often intentional corrections, indicating the robot is not performing the task as the human desires. The challenge is to enable robots to leverage these physical corrections to learn the human's true objective function during the task, and adapt their behavior accordingly, even after the human stops interacting. This is crucial for creating more intuitive and collaborative human-robot systems.

Methods

The core of the paper is a framework that formalizes physical human-robot interaction (pHRI) as a dynamical system where the robot has an unknown objective function (parameterized by $\theta$) that the human knows. Human interactions are treated as observations that help the robot infer this objective.

Key methodological contributions include:

  • Formalizing pHRI as a POMDP: The problem is framed as a Partially Observable Markov Decision Process (POMDP), where the robot's state includes its own state and its belief about the human's objective.
  • Online Learning Approximations: Exact POMDP solutions are computationally intractable for real-time applications. The authors derive approximations:
  • QMDP: Separating policy planning from belief updating.
  • MAP Estimation: Approximating the belief distribution over $\theta$ with a Maximum A Posteriori (MAP) estimate.
  • Trajectory Optimization: Moving from policy optimization to planning and tracking optimal trajectories.
  • All-at-Once Learning: An online gradient descent update rule that adjusts all feature weights of the robot's objective based on human corrections.
  • One-at-a-Time Learning: A refinement that assumes human corrections are intended to change only a single feature at a time, mitigating unintended learning from noisy interactions. This involves identifying the most likely feature to be corrected and updating only that weight.
  • Impedance Control for Tracking: The learned trajectories are tracked using a simplified impedance controller.

Datasets

Not discussed in the paper. The authors use simulated environments and user studies with a 7-DoF robotic arm (JACO2, Kinova) for evaluation.

Key Insights and Ideas

  • Physical Interaction as Communication: The fundamental insight is that physical contact with a robot is not just a disturbance but a rich source of information about the human's intent and preferences.
  • Learning from Imperfect Teachers: The framework explicitly accounts for the fact that human corrections can be noisy and imperfect, leading to the development of the "one-at-a-time" learning strategy.
  • Online Adaptation is Key: The ability for robots to learn and adapt their objectives during a task, rather than relying solely on pre-programmed behaviors or offline learning, is critical for seamless collaboration.
  • Trajectory Deformation as an Intermediate Step: Using trajectory deformations to infer an "intended trajectory" from human forces provides a tractable way to update the robot's objective.

Key Results

  • Improved Objective Performance: Simulations showed that the proposed learning method significantly outperformed impedance control (a state-of-the-art baseline) in terms of reduced regret and better alignment with the true human objective.
  • Reduced Human Effort and Interaction Time: User studies demonstrated that robots learning from pHRI required less human effort and interaction time to complete tasks compared to robots using impedance control.
  • Enhanced Human Satisfaction: Participants perceived the learning robot as more understanding, easier to interact with, and more collaborative.
  • One-at-a-Time Learning Benefits: In scenarios with noisy human corrections and multiple features to learn, the "one-at-a-time" approach showed better learning outcomes and reduced unintended learning compared to "all-at-once" learning.
  • Negligible Performance Loss: The authors found that their online learning approximation had a negligible performance difference compared to a more complete offline QMDP solution.

Limitations and Future Work

  • Feature Engineering: The current approach relies on pre-defined task-related features. Future work could explore automatic feature discovery or selection.
  • Intuitive Features: The user studies used intuitive features. Extending the approach to more complex or non-intuitive features is a direction for future research.
  • Handling Dynamical Objectives: The current framework primarily focuses on static objective parameters. Incorporating preferences about the timing or dynamics of robot actions would require further development.
  • Real-World Complexity: While the approach is demonstrated on a 7-DoF arm, scaling to even higher-dimensional systems or more complex, multi-stage tasks remains an area for exploration.
  • Human Correction Strategies: The "one-at-a-time" learning strategy assumes single-feature corrections. Further research could explore more nuanced models of human correction strategies.

Summary updated

Abstract

When a robot performs a task next to a human, physical interaction is inevitable: the human might push, pull, twist, or guide the robot. The state of the art treats these interactions as disturbanc...
BibTeX
@article{losey2022physical,
  author = {Dylan P. Losey and Andrea Bajcsy and Marcia K. O'Malley and Anca D. Dragan},
  title = {{Physical interaction as communication: Learning robot objectives online from human corrections}},
  journal = {Int. J. Robotics Res.},
  year = {2022},
  url = {https://openreview.net/forum?id=424-XQ9OHZ},
}
Figure