OFFER: Off-Environment Reinforcement Learning

Kamil Andrzej Ciosek, Shimon WhitesonAAAI 2017 · OpenReview

Abstract

Policy gradient methods have been widely applied in reinforcement learning. For reasons of safety and cost, learning is often conducted using a simulator. However, learning in simulation does not traditionally utilise the opportunity to improve learning by adjusting certain environment variables - state features that are randomly determined by the environment in a physical setting but controllable in a simulator. Exploiting environment variables is crucial in domains containing significant rare events (SREs), e.g., unusual wind conditions that can crash a helicopter, which are rarely observed under random sampling but have a considerable impact on expected return. We propose off environment reinforcement learning (OFFER), which addresses such cases by simultaneously optimising the policy and a proposal distribution over environment variables. We prove that OFFER converges to a locally optimal policy and show experimentally that it learns better and faster than a policy gradient baseline.

Introduction

In Reinforcement Learning (RL), simulators are indispensable for training agents safely and cost-effectively. However, standard simulation-based training often suffers from a "blind spot": significant rare events (SREs). If a catastrophic event—such as a helicopter-grounding gust of wind—occurs with only 0.1% probability, a standard RL agent may never encounter it frequently enough to learn a robust response.

OFFER (Off-Environment Reinforcement Learning), presented at AAAI 2017, addresses this by enabling agents to face these rare events more frequently during training without biasing the final policy.

Problem Formulation

Most RL algorithms assume the simulator's event distribution should mirror the real world. However, for high-stakes tasks like autonomous driving, sampling from the "true" distribution is inefficient; one might need to drive millions of miles to experience a single near-miss.

The authors identify environment variables—stochastic, uncontrollable features in the real world (e.g., wind speed, friction) that are controllable in a simulator. Standard policy gradient methods fail to exploit this control, leading to high-variance gradient estimates when rare, high-impact events occur. These "surprises" can destabilize the learning process with massive gradient updates.

The OFFER Method

OFFER introduces a dual-optimization approach:

  1. Primary Optimization (The Actor): Updates the agent's policy ($\theta$). To handle rewards from rare events that may be orders of magnitude larger than normal rewards, OFFER uses a stochastic variant of Newton’s method. By estimating the diagonal of the Hessian, the algorithm provides adaptive learning rates for each policy feature, preventing rare-event penalties from destabilizing training.
  2. Secondary Optimization (The Environment): This core innovation learns a proposal distribution ($\psi$) for environment variables. Instead of the "true" distribution, OFFER samples from a distribution that minimizes the variance of the policy gradient.

To maintain accuracy, Importance Sampling is used. Updates from artificially frequent "rare" events are down-weighted, ensuring the agent learns a policy optimized for the real-world distribution.

Experimental Evaluation

The authors evaluated OFFER on two main benchmarks: Modified Mountain Car (with unaliased, aliased, and 4-featured variants) and a Robotic Arm task. In the Mountain Car tasks, rare events significantly increased penalties, while in the Robotic Arm task, rare events involved potential collisions with a wall.

Experimental results across benchmarks: (a) Unaliased Mountain Car, (b) Aliased Mountain Car, (c) 4-Featured Mountain Car, and (d) Robotic Arm.

Key Results and Insights

  • Superior Performance: In Mountain Car tasks, baseline policy gradient methods showed negligible learning due to the rarity of the high-penalty events. OFFER successfully learned to handle them.
  • Adaptive Sampling: In the unaliased Mountain Car task, OFFER’s proposal distribution sampled the rare event ~80% of the time, despite its 1% real-world frequency.
  • Safety and Robustness: In the Robotic Arm task, OFFER learned a safe policy avoiding the wall, whereas the baseline failed to incorporate the wall-hit penalty effectively.
  • Convergence: The paper provides a formal proof that OFFER converges to a locally optimal policy.

Conclusion

OFFER demonstrates that "off-environment" training—learning about a target environment while training in a biased "proposal" environment—is a powerful paradigm for handling high-impact rare events. By automatically discovering these events and using second-order optimization for stability, OFFER provides a path toward more robust RL agents in complex, real-world scenarios.

Summary updated

Abstract

Policy gradient methods have been widely applied in reinforcement learning. For reasons of safety and cost, learning is often conducted using a simulator. However, learning in simulation does not traditionally utilise the opportunity to improve learning by adjusting certain environment variables - state features that are randomly determined by the environment in a physical setting but controllable in a simulator. Exploiting environment variables is crucial in domains containing significant rare events (SREs), e.g., unusual wind conditions that can crash a helicopter, which are rarely observed under random sampling but have a considerable impact on expected return. We propose off environment reinforcement learning (OFFER), which addresses such cases by simultaneously optimising the policy and a proposal distribution over environment variables. We prove that OFFER converges to a locally optimal policy and show experimentally that it learns better and faster than a policy gradient baseline.
BibTeX
@inproceedings{ciosek2017offer,
  author = {Kamil Andrzej Ciosek and Shimon Whiteson},
  title = {{OFFER: Off-Environment Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2017},
  url = {https://openreview.net/forum?id=BkNwdxWOZS},
}
Figure