Imitating careful experts to avoid catastrophic events

Jack R. P. Hanslope, Laurence AitchisonarXiv 2023 · DOI · arXiv

Abstract

Inverse Reinforcement Learning (IRL) is a powerful tool for teaching autonomous agents complex behaviors by observing experts. However, standard IRL often fails to distinguish between minor inefficiencies and catastrophic outcomes, a problem known as reward underdetermination. This paper introduces a method to resolve this ambiguity by incorporating expert "carefulness"—the tendency of humans to increase precision or reduce speed near hazards. Results in a gridworld environment demonstrate that observing carefulness allows agents to identify catastrophic risks with significantly less data than traditional methods.

Introduction

In safety-critical applications, defining an explicit reward function is often impossible. IRL addresses this by inferring rewards from expert demonstrations. Yet, if an expert avoids a dangerous zone, a standard IRL agent cannot tell if that zone is slightly inconvenient or fatal. This ambiguity can lead agents to take dangerous shortcuts. We propose that the manner in which an expert acts—specifically their level of caution—provides the missing information needed to identify catastrophic states.

Methodology: The Carefulness Signal

The researchers utilized a Gridworld environment containing a goal and a "cliff" representing a potential catastrophic event.

The Gridworld environment setup, featuring a robot agent, a goal (top right), and a red cliff zone.

In this model, an action is defined as a tuple of (direction, carefulness). Higher carefulness reduces the probability of stochastic "slips" into the cliff but incurs a higher movement cost (e.g., time or energy). The study compared a standard IRL baseline against a "Loss IRL" agent designed to minimize the difference in expected future rewards (Q-values) between expert actions and alternatives.

Results and Data Efficiency

The proposed approach showed remarkable data efficiency. While a standard Maximum Entropy (MaxEnt) baseline required 10,000 synthetic rollouts and still failed to recognize the cliff's severity, the carefulness-aware agent identified the catastrophe using only 10 human demonstrations.

The learned reward map from the carefulness-aware agent, showing high negative penalties for the cliff states, mirroring the true underlying rewards.

As shown in the recovered reward map, the agent correctly assigned massive negative values to the cliff column. In contrast, the baseline benchmark treated the cliff as a minor penalty (values around -10), illustrating the danger of ignoring behavioral nuances like caution.

Discussion and Future Work

This research highlights that carefulness is a powerful proxy for risk. By observing how experts change their behavior near ledges or hazards, agents can learn to prioritize safety. Future work will focus on scaling these concepts to high-dimensional robotic simulations where carefulness can be inferred from physical proxies like velocity or torque limits rather than explicit user inputs.

Summary updated

Abstract

RL is increasingly being used to control robotic systems that interact closely with humans. This interaction raises the problem of safe RL: how to ensure that a RL-controlled robotic system never, for instance, injures a human. This problem is especially challenging in rich, realistic settings where it is not even possible to clearly write down a reward function which incorporates these outcomes. In these circumstances, perhaps the only viable approach is based on IRL, which infers rewards from human demonstrations. However, IRL is massively underdetermined as many different rewards can lead to the same optimal policies; we show that this makes it difficult to distinguish catastrophic outcomes (such as injuring a human) from merely undesirable outcomes. Our key insight is that humans do display different behaviour when catastrophic outcomes are possible: they become much more careful. We incorporate carefulness signals into IRL, and find that they do indeed allow IRL to disambiguate undesirable from catastrophic outcomes, which is critical to ensuring safety in future real-world human-robot interactions.
BibTeX
@misc{hanslope2023imitating,
  author = {Jack R. P. Hanslope and Laurence Aitchison},
  title = {{Imitating careful experts to avoid catastrophic events}},
  year = {2023},
  eprint = {2302.01193},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2302.01193},
}
Figure