Abstract: When inferring reward functions from human behavior (be it demonstrations, comparisons, physical corrections, or e-stops), it has proven useful to model the human as making noisy-rational choices, with a "rationality coefficient" capturing how much noise or entropy we expect to see in the human behavior. Prior work typically sets the rationality level to a constant value, regardless of the type, or quality, of human feedback. However, in many settings, giving one type of feedback (e.g. a demonstration) may be much more difficult than a different type of feedback (e.g. answering a comparison query). Thus, we expect to see more or less noise depending on the type of human feedback. In this work, we advocate that grounding the rationality coefficient in real data for each feedback type, rather than assuming a default value, has a significant positive effect on reward learning. We test this in both simulated experiments and in a user study with real human feedback. We find that overestimating human rationality can have dire effects on reward learning accuracy and regret. We also find that fitting the rationality coefficient to human data enables better reward learning, even when the human deviates significantly from the noisy-rational choice model due to systematic biases. Further, we find that the rationality level affects the informativeness of each feedback type: surprisingly, demonstrations are not always the most informative---when the human acts very suboptimally, comparisons actually become more informative, even when the rationality level is the same for both. Ultimately, our results emphasize the importance and advantage of paying attention to the assumed human-rationality-level, especially when agents actively learn from multiple types of human feedback.
Introduction
In the field of Inverse Reinforcement Learning (IRL), robots aim to infer human reward functions by observing behavior. A standard approach is the Boltzmann-rational model, which posits that humans choose actions with a probability proportional to the expected reward, scaled by a "rationality coefficient" $\beta$. Traditionally, $\beta$ is treated as a fixed constant (often $\beta=1$). However, this assumption fails to account for the varying difficulty of different feedback types—such as complex demonstrations versus simple binary comparisons.
If a robot overestimates a human's rationality, it may interpret noisy or suboptimal demonstrations as precise indicators of intent, leading to significant errors in the learned reward function. This work advocates for grounding the rationality coefficient in real data for each feedback type, demonstrating that this calibration significantly improves the robustness and efficiency of reward inference.
Methodology
The proposed framework treats reward learning as a Bayesian inference problem across three core stages:
- Unified Feedback Modeling: The system models demonstrations, trajectory comparisons, and interventions (e.g., e-stops) as noisy-rational choices. This allows different feedback modalities to be processed within a single mathematical framework.
- Calibration Phase: Before tackling an unknown task, the robot undergoes a calibration phase where the human provides feedback on a task with a known reward. This data is used to calculate the maximum likelihood estimate of the human's $\beta$ for specific feedback types.
- Active Learning via Information Gain: Using the estimated $\beta$, the robot strategically selects queries (e.g., asking for a demonstration vs. a comparison) that maximize the expected Information Gain, measured by the KL-Divergence of the reward distribution.
Key Insights and Results
The research yields several critical insights into human-robot interaction through simulations and user studies:
The Asymmetry of Error
Theoretical and empirical results show that overestimating human rationality is significantly more harmful than underestimating it. When a robot assumes high rationality, it follows human errors with high confidence. Conversely, underestimating rationality leads to a more cautious, yet ultimately more accurate, learning process.
The Value of Comparisons
While demonstrations are often considered the "gold standard" for teaching robots, they are frequently noisy and difficult to provide. The study finds that when human behavior is highly suboptimal (low $\beta$), simple binary comparisons can be more informative than complex demonstrations. By learning $\beta$, the robot can autonomously shift its strategy toward these more reliable feedback types.
Robustness to Bias
Surprisingly, fitting a Boltzmann $\beta$ improves learning even when humans exhibit systematic biases, such as myopia (focusing only on immediate rewards). The coefficient acts as a buffer, signaling to the robot that the human's choices should not be trusted implicitly, thereby mitigating the impact of non-rational behavior.
Conclusion
Grounding the rationality coefficient in empirical data is essential for effective reward learning from multiple feedback types. By moving away from fixed assumptions of human perfection, robots can better navigate the noise inherent in human behavior, choosing the most informative ways to ask for help and ultimately aligning more closely with human intent. Future work should address the calibration burden and explore dynamic models of rationality that account for human fatigue or learning over time.
Summary updated