Survival Instinct in Offline Reinforcement Learning

Anqi Li, Dipendra Misra, Andrey Kolobov, Ching-An Cheng

Abstract

We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of pessimism in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a "survival instinct", i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class—which may not even contain the true reward—we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. Our results suggest a new paradigm for RL, whereby an agent is nudged to learn a desirable behavior with imperfect reward but purposely biased data coverage.

Introduction

Reinforcement Learning (RL) is notoriously sensitive to reward engineering. In online settings, agents are expert "reward hackers"; if a reward function is slightly misaligned with the intended goal, the agent will exploit that discrepancy, often leading to catastrophic or nonsensical behaviors. However, a strange anomaly has emerged in the field of Offline RL, where agents learn from fixed, pre-collected datasets. Researchers have observed that offline agents can learn to solve complex tasks even when the rewards are set to zero, randomized, or even negated.

This paper investigates this "doubly robust" nature of offline RL. We propose that the success of these agents under "garbage" reward signals is not a fluke, but the result of a survival instinct—a byproduct of the interplay between algorithmic pessimism and implicit biases within the training data.

The Mechanism: Pessimism and Data Bias

The survival instinct arises from two primary factors:

1. Algorithmic Pessimism

Modern offline RL algorithms (such as CQL, IQL, or ATAC) are designed to be pessimistic. Because they cannot explore the environment to verify the outcomes of new actions, they penalize transitions that lead to states or actions not well-represented in the training dataset. Effectively, the agent views leaving the "support" of the data as the worst possible outcome.

2. Implicit Data Bias

Real-world datasets are rarely uniform. They often contain implicit biases that correlate "staying within the data" with "performing the task correctly":

  • Length Bias: In many robotics tasks, a trajectory ends when the robot fails (e.g., a robot falls over). Consequently, successful trajectories are naturally longer and more prevalent in the dataset than failure states.
  • Intervention Bias: In safety-critical domains, a human operator might intervene and stop a sequence before a collision occurs, ensuring the dataset only contains safe boundaries.

Theoretical Framework: RL as a Constrained MDP

We formalize this behavior by showing that a pessimistic offline RL agent implicitly solves a Constrained Markov Decision Process (CMDP). Instead of purely maximizing a reward signal, the agent seeks to maximize the reward while remaining within the data support.

If the data support is "positively biased"—meaning that staying within the distribution is synonymous with task success—the agent develops an incentive to survive within that distribution for as long as possible. This survival instinct can override even a negative reward signal, as the penalty for "dying" (leaving the data support) is perceived as more severe than the penalty of the negative reward labels.

Empirical Evidence

The Success of "Zero-Reward" Learning

In evaluations using the D4RL benchmark, specifically the Hopper task, the ATAC algorithm trained on zero rewards (where every transition was labeled 0) outperformed the original data collection policy and standard Behavior Cloning (BC). In several instances, it matched the performance of agents trained on the true, expert-engineered rewards. The agent learned that to "survive" in the dataset for 1,000 steps, it had to keep hopping; falling over would lead to an out-of-distribution (OOD) state, which the pessimism term heavily penalized.

Robustness in Meta-World

In Meta-World manipulation tasks (e.g., opening a drawer), agents were trained on negative rewards—labels that should theoretically encourage the agent to fail as quickly as possible. Despite this, the agents achieved high success rates. Because the only way to remain within the data support for the full episode duration was to complete the task, the survival instinct forced the agent to succeed.

When the Instinct Fails: HalfCheetah

The theory was further validated by its failures. In the HalfCheetah environment, trajectories are always a fixed 1,000 steps regardless of whether the robot runs forward, backward, or flips over. Because there is no length bias, the survival instinct has no "death" to avoid. In this environment, agents trained on wrong rewards failed, confirming that the survival instinct depends entirely on the presence of data bias.

Conclusion and Future Directions

Our findings suggest that the performance of offline RL on current benchmarks may be partially attributed to the survival instinct rather than the algorithms' ability to optimize rewards. This has significant implications for how we interpret RL results and design future benchmarks.

Furthermore, this research points toward a new paradigm for RL: Data-Centric Nudging. Instead of spending months engineering complex reward functions, we may be able to achieve desired behaviors by carefully biasing data collection or filtering datasets to create a support that only contains safe or successful behaviors. By leveraging the survival instinct, we can make RL more robust and easier to deploy in real-world scenarios where reward signals are difficult to define.

Summary updated

Abstract

We present a novel observation about the behavior of offline reinforcement learning (RL) algorithms: on many benchmark datasets, offline RL can produce well-performing and safe policies even when trained with "wrong" reward labels, such as those that are zero everywhere or are negatives of the true rewards. This phenomenon cannot be easily explained by offline RL's return maximization objective. Moreover, it gives offline RL a degree of robustness that is uncharacteristic of its online RL counterparts, which are known to be sensitive to reward design. We demonstrate that this surprising robustness property is attributable to an interplay between the notion of pessimism in offline RL algorithms and certain implicit biases in common data collection practices. As we prove in this work, pessimism endows the agent with a "survival instinct", i.e., an incentive to stay within the data support in the long term, while the limited and biased data coverage further constrains the set of survival policies. Formally, given a reward class -- which may not even contain the true reward -- we identify conditions on the training data distribution that enable offline RL to learn a near-optimal and safe policy from any reward within the class. We argue that the survival instinct should be taken into account when interpreting results from existing offline RL benchmarks and when creating future ones. Our empirical and theoretical results suggest a new paradigm for RL, whereby an agent is nudged to learn a desirable behavior with imperfect reward but purposely biased data coverage.
BibTeX
@inproceedings{li2023survival,
  author = {Anqi Li and Dipendra Misra and Andrey Kolobov and Ching-An Cheng},
  title = {{Survival Instinct in Offline Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=shePL2nbwl},
}
Figure