What Is It You Really Want of Me? Generalized Reward Learning with Biased Beliefs about Domain Dynamics

Ze Gong, Yu Zhang

Abstract

Traditional reward learning assumes that humans have a perfect understanding of a robot's capabilities. This paper introduces GeReL, a framework that acknowledges humans often hold biased beliefs about how a robot works, allowing the system to simultaneously infer both the human's true preferences and their misconceptions about the robot's dynamics.

Introduction

When teaching robots through feedback—such as ratings or demonstrations—standard algorithms like Inverse Reinforcement Learning (IRL) assume the human provider has a perfect "mental model" of the robot’s physical constraints (domain dynamics). In reality, non-expert users are often biased. A user might assume a humanoid robot is more agile than a wheeled one, or that a vacuum robot can navigate surfaces it actually cannot. If a user provides a low rating because a robot failed a task it was physically incapable of performing, standard IRL mistakenly concludes the user does not value that task. This leads to a fundamental misunderstanding of human intent.

The GeReL Framework

The authors propose GeReL (Generalized Reward Learning with biased beliefs about domain dynamics). Instead of treating the robot's dynamics as a known constant, GeReL treats the human's belief about those dynamics as a latent variable to be discovered.

The GeReL framework models the robot's inference of the human's latent reward function and biased beliefs about dynamics.

Technical Approach

  • Bayesian Framework: The model seeks the posterior distribution of reward weights ($w$) and parameters ($\Theta$) governing the human's belief of transition probabilities.
  • Variational Inference: To approximate the intractable true posterior, the authors use Black-Box Variational Inference (BBVI), optimizing the Evidence Lower Bound (ELBO) via stochastic gradient descent.
  • Human Rating Model: Human ratings are modeled as proportional to the alignment between the robot's actual trajectory and what the human thinks the optimal trajectory should be, given their biased beliefs.

Experimental Evaluation

GeReL was evaluated across three distinct domains:

  1. Simulated Navigation: A $7 \times 7$ grid world with synthetic humans programmed with specific biases (e.g., reversed directional beliefs).
  2. Coffee Robot Domain: A benchmark MDP where a robot delivers coffee under varying weather conditions.
  3. User Study: 20 participants on Amazon Mechanical Turk interacted with two robot types (humanoid vs. mobile) to induce natural biases regarding stability and capability.

Key Results

GeReL demonstrated superior performance in recovering true human intent by accounting for the "lens" of human belief through which feedback is filtered.

Comparison of reward recovery across different methods. GeReL closely matches the ground truth compared to baselines like MaxEnt-IRL.
  • Quantitative Recovery: In grid-world simulations, GeReL recovered the true reward function with an $L_2$ distance of ~12, significantly outperforming MaxEnt-IRL (~23-28), which failed to account for biased beliefs.
  • Appearance-Induced Bias: The user study confirmed that robot form factor significantly biases human expectations. Participants expected the humanoid to be more capable than the mobile robot, despite identical ground-truth dynamics. GeReL correctly identified that users valued the robot staying dry, whereas baselines concluded users were indifferent to the robot's state.

Conclusion and Future Work

By breaking the symmetry between rewards and dynamics through informative priors, GeReL provides a more robust path for human-robot interaction. Future work may explore "closing the loop"—determining whether a robot should act to correct a human's biased belief through explanation or simply adapt its behavior to accommodate the user's misconceptions.

Summary updated

BibTeX
@inproceedings{gong2020what,
  author = {Ze Gong and Yu Zhang},
  title = {{What Is It You Really Want of Me? Generalized Reward Learning with Biased Beliefs about Domain Dynamics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i03.5630},
}
Figure