Replay For Safety

Liran Szlak, Ohad ShamirarXiv 2021 · arXiv

Abstract

Standard reinforcement learning (RL) agents are typically designed to be risk-neutral, aiming to maximize the expected value of cumulative rewards. However, in safety-critical environments, this objective can lead to catastrophic failures. This paper introduces a novel framework for training risk-averse agents by strategically biasing their experience replay. By forcing agents to disproportionately "relive" memories of high-variance states and negative outcomes, the proposed method induces cautious behavior without requiring modifications to the core optimization objectives or reward functions.

Introduction

In the pursuit of maximizing average returns, standard RL agents often ignore the variance of those returns. An autonomous vehicle might choose a route that is five minutes faster on average but carries a non-zero probability of a fatal collision. Traditional "Safe RL" approaches attempt to mitigate this by adding variance penalties to reward functions or employing complex, computationally expensive "worst-case" optimization criteria. These methods often require significant domain expertise or perfect prior knowledge of environmental risks.

This research explores an alternative: Can safety be achieved simply by changing how an agent remembers its past? By treating experience replay as a perceptual lens rather than a neutral data store, the authors demonstrate that agents can be "taught" caution through biased memory sampling.

Methodology

The authors focus on Q-learning, a fundamental RL algorithm, and its use of an Experience Replay buffer. Their approach consists of two primary components: a theoretical framework for convergence and a specific sampling heuristic.

1. Convergent Replay Framework

A significant challenge in biasing training data is ensuring the algorithm still converges to a stable policy. The authors provide a mathematical proof demonstrating that Q-learning remains convergent even under non-uniform sampling schemes, provided the sampling distribution eventually stabilizes into a consistent pattern. This allows for the intentional distortion of the agent's experience without breaking the underlying learning mechanics.

2. Variance Prioritized Replay (VPR)

The core of the proposed method is a specific sampling scheme designed to instill risk-aversion. Instead of sampling experiences uniformly, the agent follows a two-step prioritization process:

  • High-Variance Prioritization: The agent prioritizes replaying state-action pairs that have exhibited high reward volatility. This ensures the agent focuses its learning capacity on the most unpredictable parts of the environment.
  • Negative Outcome Bias: Within these volatile states, the agent uses a Boltzmann-like distribution (controlled by a sensitivity parameter $\beta$) to give higher priority to the most negative outcomes encountered.

By over-representing these "scary" memories during training, the agent's perceived value (Q-value) for risky actions decreases. Effectively, the agent learns to avoid these states not because the reward function changed, but because its internal model of the world is biased toward the worst-case scenarios it has experienced.

Experimental Evaluation

The researchers evaluated the Variance Prioritized Replay (VPR) method in two discrete "Grid World" environments designed to test risk sensitivity:

  1. Reward Variance Grid: A 4x5 grid where the agent must choose between a short, volatile path (60% chance of +100 reward, 40% chance of -100) and a longer, stable path.
  2. The Cliff Environment: A 4x6 grid where actions have a 10% failure rate. A single misstep can push the agent off a "cliff," resulting in a significant penalty (-12).

Key Results

  • Safety without Penalty: In the Reward Variance Grid, VPR successfully trained agents to prefer the longer, safer path, whereas standard Q-learning consistently chose the risky shortcut.
  • Performance against Baselines: In the Cliff Environment, the VPR agent outperformed existing Safe RL baselines, such as the Mihatsch & Neuneier algorithm. While baselines often struggled to balance optimism and pessimism in the presence of transition noise, the biased replay agent consistently maintained a safe distance from the cliff edge.
  • Theoretical Validation: The experiments confirmed that biased replay schemes do not prevent the agent from reaching a stable, convergent policy.

Discussion and Biological Parallels

The authors suggest that this approach mirrors biological learning mechanisms. Humans and animals do not treat all memories equally; high-arousal or fearful experiences are often "replayed" more intensely in the brain. This evolutionary trait serves as a survival mechanism, ensuring that dangerous mistakes are not repeated. By implementing a similar bias in RL, we can create agents that are naturally more cautious.

Limitations and Future Work

While promising, the current study is limited to the tabular setting, where states and actions are discrete. Future research is required to determine how these biased sampling schemes interact with Deep RL and function approximation, where neural networks might react unpredictably to highly skewed data distributions. Additionally, the "level" of safety is currently dependent on manual tuning of hyperparameters like $\beta$ (negative bias) and $v$ (replay ratio). Automating the tuning of these parameters to match specific safety requirements remains an open challenge.

Conclusion

Replay For Safety demonstrates that risk-aversion can be an emergent property of how an agent processes its history. By shifting the focus from complex optimization math to the strategic management of experience data, this research provides a simpler, more intuitive pathway toward developing safe and reliable autonomous systems.

Summary updated

Abstract

Experience replay \citep{lin1993reinforcement, mnih2015human} is a widely used technique to achieve efficient use of data and improved performance in RL algorithms. In experience replay, past transitions are stored in a memory buffer and re-used during learning. Various suggestions for sampling schemes from the replay buffer have been suggested in previous works, attempting to optimally choose those experiences which will most contribute to the convergence to an optimal policy. Here, we give some conditions on the replay sampling scheme that will ensure convergence, focusing on the well-known Q-learning algorithm in the tabular setting. After establishing sufficient conditions for convergence, we turn to suggest a slightly different usage for experience replay - replaying memories in a biased manner as a means to change the properties of the resulting policy. We initiate a rigorous study of experience replay as a tool to control and modify the properties of the resulting policy. In particular, we show that using an appropriate biased sampling scheme can allow us to achieve a \emph{safe} policy. We believe that using experience replay as a biasing mechanism that allows controlling the resulting policy in desirable ways is an idea with promising potential for many applications.
BibTeX
@misc{szlak2021replay,
  author = {Liran Szlak and Ohad Shamir},
  title = {{Replay For Safety}},
  year = {2021},
  eprint = {2112.04229},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.04229},
}
Figure