Utilizing Human Feedback for Primitive Optimization in Wheelchair Tennis

Arjun Krishna, Zulfiqar Zaidi, Letian Chen, Rohan R. Paleja, Esmaeil Seraj, Matthew C. GombolayarXiv 2022 · DOI · OpenReview

TL;DR

This paper tackles the challenge of teaching agile mobile robots complex, high-speed tasks like playing wheelchair tennis. They extend a probabilistic movement primitive (ProMP) framework to enable a robot to safely execute tennis strokes and, crucially, refine these learned movements using human evaluative feedback. This work paves the way for more adaptable and trustworthy agile robots in dynamic environments.

What Problem Does This Paper Address?

Developing robots capable of performing agile, high-speed maneuvers, such as those required in sports, is a significant challenge. These robots need to be precise, responsive, and safe to gain human trust. While learning from demonstrations (LfD) has shown promise, applying it to dynamic, whole-body movements on mobile platforms, especially in complex scenarios like wheelchair tennis, remains difficult. Existing methods often fall short in adapting to real-world complexities, such as variations in ball trajectory and the robot's own physical limitations, leading to suboptimal performance. This paper specifically addresses the need for a flexible and adaptive controller for an agile mobile manipulator to perform ground strokes in wheelchair tennis, and introduces a method to improve these learned behaviors through human input.

Methods

The core of the paper's contribution lies in extending the Probabilistic Movement Primitive (ProMP) framework. ProMP is a technique that compactly represents a distribution over robot trajectories, allowing for the learning of common movement patterns and their variations. The authors make two key advancements:

  1. Safe Execution of Learned Primitives: They adapt the ProMP framework for an agile mobile manipulator (a 7-DOF arm on a wheelchair). A critical aspect is ensuring the safety of high-speed strokes. This is achieved by implementing a "clipped inverse kinematics" approach. When conditioning the ProMP to hit a specific point, the system clips the resulting joint commands to stay within pre-specified limits, preventing potentially unsafe or infeasible solutions. They also incorporate a state-machine for the stroke controller that allows the wheelchair to continuously adjust its position for better stroke execution.
  2. Online Primitive Refinement with Human Feedback: To improve the learned primitives beyond initial demonstrations, they propose an iterative refinement procedure. A human evaluator provides scalar feedback on the quality of executed trajectories. This feedback is then used to compute importance weights for each trajectory. These weighted trajectories are used to update the ProMP parameters via an Expectation-Maximization (EM) procedure, effectively shaping the learned movement distribution based on human preferences. This approach is framed as a form of interactive shaping, similar to TAMER algorithms.

Datasets

The paper does not explicitly mention a large, pre-existing dataset for training the initial ProMP. Instead, they initialize the ProMP parameters by training it on a dataset of successful demonstrations obtained from a manually engineered stroke. For evaluating the refinement procedure, they collect data by running the base primitive and recording joint states. Human evaluators then provide feedback on these executed trajectories.

Key Insights and Ideas

  • Human feedback is crucial for refining complex robotic skills: While initial demonstrations provide a good starting point, human evaluators can identify subtle issues and preferences that are hard to capture otherwise, leading to improved performance.
  • Safety is paramount for agile robots: Simply learning to hit the ball isn't enough; the robot must do so without self-collision or executing dangerous movements. The "clipped inverse kinematics" is a practical solution for ensuring safe execution on a real-world robotic system.
  • Wheelchair tennis is a challenging benchmark: The combination of a mobile base, a high-speed arm, and the need for precise timing makes this task significantly more complex than stationary robotic manipulation tasks.
  • Iterative refinement can lead to better performance, even with limited data: The proposed method shows promise in improving the average reward, suggesting that even with a moderate number of feedback-driven refinements, the robot's performance can be enhanced.

Key Results

The paper reports on the performance of the base primitive and the refined primitive over a consecutive run of 10 balls.

  • Base Primitive: Achieved a Hit Rate of 60%, Success Rate of 40%, and an Average Reward of 0.75.
  • Refined Primitive (20 trajectories): Achieved a Hit Rate of 40%, Success Rate of 40%, and an Average Reward of 0.85.
  • Refined Primitive (50 trajectories): Achieved a Hit Rate of 50%, Success Rate of 40%, and an Average Reward of 0.85.

While the success rate remained constant, the average reward increased with refinement, indicating that the strokes became "good hits" rather than "misses by small margins." The hit rate also showed improvement with 50 trajectories compared to 20.

Limitations and Future Work

The authors acknowledge that they did not observe significant improvements in success rates with their current fine-tuning approach. They suggest that future work could explore different reward designs to better capture desired outcomes and study how the choice of reward impacts the learned primitive.

Further future work directions include:

  • Developing methods to ensure safer behavior and increase task performance more broadly.
  • Investigating the relationship between their work and Active Learning with Human Feedback, Reinforcement Learning of robot skills, and Human-Robot Interaction.
  • Exploring various feedback paradigms, such as natural language, kinesthetic teaching, and third-person demonstrations, to further enhance robot motion learning.

Summary updated

Abstract

Agile robotics presents a difficult challenge with robots moving at high speeds requiring precise and low-latency sensing and control. Creating agile motion that accomplishes the task at hand while being safe to execute is a key requirement for agile robots to gain human trust. This requires designing new approaches that are flexible and maintain knowledge over world constraints. In this paper, we consider the problem of building a flexible and adaptive controller for a challenging agile mobile manipulation task of hitting ground strokes on a wheelchair tennis robot. We propose and evaluate an extension to work done on learning striking behaviors using a probabilistic movement primitive (ProMP) framework by (1) demonstrating the safe execution of learned primitives on an agile mobile manipulator setup, and (2) proposing an online primitive refinement procedure that utilizes evaluative feedback from humans on the executed trajectories.
BibTeX
@misc{krishna2022utilizing,
  author = {Arjun Krishna and Zulfiqar Zaidi and Letian Chen and Rohan R. Paleja and Esmaeil Seraj and Matthew C. Gombolay},
  title = {{Utilizing Human Feedback for Primitive Optimization in Wheelchair Tennis}},
  year = {2022},
  eprint = {2212.14403},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.14403},
}
Figure