The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types

Gaurav R. Ghosal, Matthew Zurek, Daniel S. Brown, Anca D. Dragan

Abstract

When inferring reward functions from human behavior, researchers typically model humans as "noisy-rational" agents using a fixed rationality coefficient ($\beta$). However, human proficiency varies significantly across different feedback types, such as demonstrations versus simple comparisons. This work advocates for grounding the rationality coefficient in real data for each feedback type. Through simulated experiments and a user study, we demonstrate that overestimating human rationality leads to significant learning errors. Conversely, fitting $\beta$ to human data improves reward learning accuracy and enables more efficient active learning by identifying which feedback types are most informative given a specific user's noise level.

Introduction

In Inverse Reinforcement Learning (IRL), the Boltzmann-rational model is the standard framework for recovering a human's underlying reward function. It assumes the probability of an action is proportional to the exponent of its reward, scaled by $\beta$. While most systems treat $\beta$ as a fixed constant (often $\beta=1$), humans are rarely equally rational across all tasks. Providing a complex physical demonstration is cognitively more demanding than choosing between two options. If a robot assumes high rationality when a human is actually noisy, it confidently learns incorrect rewards. This paper explores the impact of explicitly modeling and fitting $\beta$ for different feedback modalities.

Methods

The authors propose a two-step framework for robust reward learning:

  1. Rationality Fitting: A "calibration phase" is introduced where a human provides feedback on a task with a known reward function. Using Maximum Likelihood Estimation (MLE), the robot fits an estimate ($\hat{\beta}$) for that specific individual and feedback type.
  2. Active Learning via Information Gain: To learn an unknown reward, the robot employs Bayesian experimental design. It uses the fitted $\hat{\beta}$ to calculate the Expected Information Gain for various query types (e.g., demonstrations, comparisons, or e-stops), allowing it to select the most informative interaction.

Key Insights: The Informativeness Crossover

A critical finding of this research is that the most "information-rich" feedback type depends entirely on the human's rationality level. While demonstrations provide high-dimensional data, they are prone to significant noise if the human finds the task difficult.

Figure 1: Comparison of reward learning outcomes when using a default rationality coefficient versus a learned one. The learned model correctly identifies that comparisons are more informative than noisy demonstrations.

As shown in the figure above, when a robot assumes a default high rationality (left), it requests a demonstration that results in a noisy path, leading to the incorrect conclusion that the human wants to hit obstacles. By learning that the human is noisy (right), the robot instead requests a comparison, which restricts the choice set and allows for accurate reward inference even with the same level of human noise.

Experimental Results

The approach was validated across three environments: a simulated Gridworld, a car driving simulator, and a user study with 7 participants.

  • Asymmetry of Error: It is far more dangerous to overestimate human rationality than to underestimate it. Overestimation leads to overfitting on human mistakes, while underestimation maintains a broader, safer belief about the reward.
  • User Study Validation: Fitting $\beta$ for demonstrations led to significantly better reward inference ($p=0.02$).
  • Active Learning Efficiency: When the robot actively chose between feedback types using the fitted $\beta$, performance gains were massive ($p=0.0001$). The system learned to "distrust" noisy demonstrations in favor of easier-to-provide comparisons.
  • Bias Mitigation: Fitting $\beta$ acts as an effective buffer against systematic human biases, such as myopia, allowing the robot to remain effective even when the human deviates from the Boltzmann model.

Conclusion

This work emphasizes that the assumed level of human rationality is a primary factor in the success of reward learning. By moving away from fixed defaults and grounding $\beta$ in data, agents can better navigate the trade-offs between different feedback types, leading to safer and more accurate human-robot alignment. Future work will explore joint inference of rewards and rationality levels to eliminate the need for a separate calibration phase.

Summary updated

Abstract

When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of the type, or quality, of human feedback. However, in many settings, giving one type of feedback (e.g. a demonstration) may be much more difficult than a different type of feedback (e.g. answering a comparison query). Thus, we expect to see more or less noise depending on the type of human feedback. In this work, we advocate that grounding the rationality coefficient in real data for each feedback type, rather than assuming a default value, has a significant positive effect on reward learning. We test this in both simulated experiments and in a user study with real human feedback. We find that overestimating human rationality can have dire effects on reward learning accuracy and regret. We also find that fitting the rationality coefficient to human data enables better reward learning, even when the human deviates significantly from the noisy-rational choice model due to systematic biases. Further, we find that the rationality level affects the informativeness of each feedback type: surprisingly, demonstrations are not always the most informative -- when the human acts very suboptimally, comparisons actually become more informative, even when the rationality level is the same for both. Ultimately, our results emphasize the importance and advantage of paying attention to the assumed human-rationality level, especially when agents actively learn from multiple types of human feedback.
BibTeX
@inproceedings{ghosal2023effect,
  author = {Gaurav R. Ghosal and Matthew Zurek and Daniel S. Brown and Anca D. Dragan},
  title = {{The Effect of Modeling Human Rationality Level on Learning Rewards from Multiple Feedback Types}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=jekTXNIDOX},
}
Figure