Introduction
Specifying every single thing a robot shouldn't do is an impossible task. This paper introduces Reward Learning by Simulating the Past (RLSP), an algorithm that allows robots to infer human preferences simply by observing the current state of their environment, under the clever insight that the world we live in is already "optimized" for what we want.
The Side Effects Problem
In Reinforcement Learning (RL), we usually tell a robot what to do via a reward function. However, humans are notoriously bad at being exhaustive. If you tell a robot to "go to the purple door" but forget to say "don't smash the expensive vases on the way," a standard RL agent will happily trample the vases to find the shortest path. This is known as the side effects problem.
Traditionally, researchers have tried to solve this by looking at human actions (demonstrations) or asking for feedback. But these methods are data-hungry and time-consuming. This paper addresses a fundamental gap: we have been ignoring a massive, "free" source of information—the current state of the world itself. If a robot is deployed in a room where the vases are unbroken, it should be able to infer that the person living there probably cares about keeping them that way.
Reward Learning by Simulating the Past (RLSP)
The authors propose RLSP, which treats the current state of the world ($s_0$) as the "final state" of a human's trajectory.
- Inverse Reinforcement Learning (IRL) from a Single State: RLSP is a variant of Maximum Causal Entropy IRL. While standard IRL requires full demonstrations (sequences of actions), RLSP works with just one state.
- Simulating the Past: The algorithm assumes a human "Alice" was acting in the environment for some time $T$ before the robot arrived. It searches for a reward function that makes the observed state $s_0$ the most likely outcome of Alice’s behavior.
- Mathematical Formulation: It marginalizes over all possible past trajectories that could have led to the current state. By using gradient ascent, it finds reward parameters $\theta$ that maximize the likelihood of the world ending up exactly as the robot found it.
- Combining Rewards: Once the robot infers Alice's implicit preferences ($\theta_{Alice}$), it combines them with the explicitly specified task reward ($\theta_{spec}$) to create a final, safe policy.
Environments and Key Results
The authors evaluated RLSP on a suite of custom-designed Gridworld environments meant to test specific safety and preference properties.
- Room with vase: Can the robot avoid a fragile object not mentioned in the reward? RLSP successfully reached the goal without breaking the vase, whereas the standard agent smashed it.
- Toy train: Can the robot distinguish between changes it causes vs. changes the environment causes (a moving train)?
- Apple collection: Can the robot infer an active task (harvesting) without any specified reward? Given zero specified reward, the robot saw the basket was full in $s_0$ and correctly inferred it should continue harvesting apples.
- Batteries: Can the robot identify "desirable" side effects (keeping a train powered)? Unlike "impact penalty" baselines that mindlessly try to keep everything the same, RLSP understood that "using up" a battery to keep a train running was a desirable side effect.
- Far away vase: A "failure case" environment where the side effect is outside the human's normal path.
Key Insights
- The "Optimization" Insight: The most powerful idea here is that the initial state of a robot's deployment isn't "neutral"—it is a curated result of human effort. Inaction is often a choice to preserve a desired state.
- Beyond "Don't Change Anything": Previous safety methods tried to penalize any change to the environment. RLSP is smarter; it realizes that some changes (like a full basket of apples) are actually desired by humans, while others (broken vases) are not.
- Robustness to the Past: Surprisingly, the robot doesn't need to know exactly where the human started. Even with a "uniform prior" (assuming the human could have started anywhere), the algorithm is quite effective at identifying what features humans tend to preserve.
Limitations and Future Work
- The "Unseen" Limitation: If a side effect is located somewhere a human would never go (like a vase in a far corner), RLSP won't realize it's important because the human's "silence" regarding that vase provides no information.
- Scaling to Reality: The current work assumes known dynamics and hand-coded features. Moving to "pixels-to-actions" in complex, 3D environments with unknown physics is the major next step.
- The Horizon $T$: The algorithm requires an assumption of how long the human has been acting ($T$). If $T$ is set incorrectly, the robot might misinterpret the human's level of "laziness" or "efficiency."
- Non-Human Optimization: The paper notes that some things we like (like oxygen in the atmosphere) aren't the result of human optimization. RLSP wouldn't know to preserve those.
Summary updated