Abstract
Inverse Reinforcement Learning (IRL) is a powerful tool for teaching autonomous agents complex behaviors by observing experts. However, standard IRL often fails to distinguish between minor inefficiencies and catastrophic outcomes, a problem known as reward underdetermination. This paper introduces a method to resolve this ambiguity by incorporating expert "carefulness"—the tendency of humans to increase precision or reduce speed near hazards. Results in a gridworld environment demonstrate that observing carefulness allows agents to identify catastrophic risks with significantly less data than traditional methods.
Introduction
In safety-critical applications, defining an explicit reward function is often impossible. IRL addresses this by inferring rewards from expert demonstrations. Yet, if an expert avoids a dangerous zone, a standard IRL agent cannot tell if that zone is slightly inconvenient or fatal. This ambiguity can lead agents to take dangerous shortcuts. We propose that the manner in which an expert acts—specifically their level of caution—provides the missing information needed to identify catastrophic states.
Methodology: The Carefulness Signal
The researchers utilized a Gridworld environment containing a goal and a "cliff" representing a potential catastrophic event.
In this model, an action is defined as a tuple of (direction, carefulness). Higher carefulness reduces the probability of stochastic "slips" into the cliff but incurs a higher movement cost (e.g., time or energy). The study compared a standard IRL baseline against a "Loss IRL" agent designed to minimize the difference in expected future rewards (Q-values) between expert actions and alternatives.
Results and Data Efficiency
The proposed approach showed remarkable data efficiency. While a standard Maximum Entropy (MaxEnt) baseline required 10,000 synthetic rollouts and still failed to recognize the cliff's severity, the carefulness-aware agent identified the catastrophe using only 10 human demonstrations.
As shown in the recovered reward map, the agent correctly assigned massive negative values to the cliff column. In contrast, the baseline benchmark treated the cliff as a minor penalty (values around -10), illustrating the danger of ignoring behavioral nuances like caution.
Discussion and Future Work
This research highlights that carefulness is a powerful proxy for risk. By observing how experts change their behavior near ledges or hazards, agents can learn to prioritize safety. Future work will focus on scaling these concepts to high-dimensional robotic simulations where carefulness can be inferred from physical proxies like velocity or torque limits rather than explicit user inputs.
Summary updated