Preferences Implicit in the State of the World

Rohin Shah, Dmitrii Krasheninnikov, Jordan Alexander, Pieter Abbeel, Anca D. Dragan

Abstract

Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized.

Introduction

Reinforcement learning (RL) agents are notoriously literal. If you tell a robot to "get to the door" and forget to mention "don't smash the expensive Ming vase," the robot might take the shortest path right through the porcelain. This is the reward specification problem: it is incredibly difficult to exhaustively list everything a robot shouldn't do.

Traditionally, researchers try to solve this by watching humans act (Inverse Reinforcement Learning) or asking for feedback. However, these methods require active human participation. This paper identifies a "free" and overlooked source of information: the initial state of the environment. Because humans have been acting in the world long before the robot arrived, the current state is already a local optimum of human preferences. If the vases are currently unbroken, it is a safe bet that humans want them to stay that way.

Reward Learning by Simulating the Past (RLSP)

The authors propose an algorithm called Reward Learning by Simulating the Past (RLSP). The core idea is to treat the robot’s starting state ($s_0$) as the final state of a hypothetical human trajectory that began at some earlier time ($s_{-T}$).

RLSP infers rewards by assuming the current state $s_0$ is the result of human actions from a past state $s_{-T}$.

RLSP builds on Maximum Causal Entropy IRL, but with a twist: instead of having a full demonstration of actions, it only has the destination state. The algorithm models a "Boltzmann-rational" human agent who chooses actions to maximize reward. It considers various possible reward functions and calculates which ones would most likely lead a human to end up in the observed state $s_0$ starting from a past state $s_{-T}$. By marginalizing over all possible past trajectories, the robot computes a Maximum Likelihood Estimate (MLE) of the human's reward function. The robot then combines this inferred "human" reward with its specifically assigned task reward to navigate the world safely.

Evaluation and Results

The researchers evaluated RLSP on a suite of custom-designed gridworld environments:

  • Room with Vase: Avoiding side effects like breaking a vase.
  • Toy Train: Distinguishing between agent-caused changes and natural environment dynamics.
  • Apple Collection: Inferring active tasks, such as harvesting apples.
  • Batteries: Recognizing desirable side effects, like using resources to maintain a system.
Comparison of trajectories between a literal task-specified policy ($\pi_{spec}$) and the RLSP policy ($\pi_{RLSP}$) across various environments.

In the "Room with Vase" scenario, RLSP successfully inferred a negative reward for broken vases, allowing the robot to reach its goal via a longer path that kept the vase intact—even though "don't break vases" was never explicitly coded. RLSP outperformed standard "impact penalties" which penalize all changes. For example, in the Apple environment, impact penalties prevented the robot from harvesting, while RLSP recognized that harvesting was a preferred state. The algorithm also proved relatively robust to the "planning horizon" ($T$); even if the robot's guess of how long the human had been acting was slightly off, it still captured the core preferences.

Discussion and Future Work

The most powerful insight of this work is that the "inaction baseline" (doing nothing) is only a good strategy because the world is already optimized. RLSP explains why this works and allows the robot to be more flexible than just "don't change anything." It can infer both static preferences (don't break vases) and active ones (harvest apples). If a state is "hard to reach" but currently exists, the robot infers that the human must have valued the effort to get there.

However, limitations remain. RLSP can only infer preferences for things the human has actually interacted with. If a vase is in a far corner where the human never goes, the robot won't see "unbroken-ness" as a result of human effort. Furthermore, the current model assumes known dynamics and hand-coded features. Future work aims to integrate RLSP with deep RL techniques to handle more complex, high-dimensional environments and raw pixel inputs.

Summary updated

Abstract

Reinforcement learning (RL) agents optimize only the features specified in a reward function and are indifferent to anything left out inadvertently. This means that we must not only specify what to do, but also the much larger space of what not to do. It is easy to forget these preferences, since these preferences are already satisfied in our environment. This motivates our key insight: when a robot is deployed in an environment that humans act in, the state of the environment is already optimized for what humans want. We can therefore use this implicit preference information from the state to fill in the blanks. We develop an algorithm based on Maximum Causal Entropy IRL and use it to evaluate the idea in a suite of proof-of-concept environments designed to show its properties. We find that information from the initial state can be used to infer both side effects that should be avoided as well as preferences for how the environment should be organized. Our code can be found at https://github.com/HumanCompatibleAI/rlsp.
BibTeX
@inproceedings{shah2019preferences,
  author = {Rohin Shah and Dmitrii Krasheninnikov and Jordan Alexander and Pieter Abbeel and Anca D. Dragan},
  title = {{Preferences Implicit in the State of the World}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
  url = {https://openreview.net/forum?id=PlJlHlS9cf},
}

Loading related content...

Figure