Abstract
Policy gradient methods have been widely applied in reinforcement learning. For reasons of safety and cost, learning is often conducted using a simulator. However, learning in simulation does not traditionally utilise the opportunity to improve learning by adjusting certain environment variables - state features that are randomly determined by the environment in a physical setting but controllable in a simulator. Exploiting environment variables is crucial in domains containing significant rare events (SREs), e.g., unusual wind conditions that can crash a helicopter, which are rarely observed under random sampling but have a considerable impact on expected return. We propose off environment reinforcement learning (OFFER), which addresses such cases by simultaneously optimising the policy and a proposal distribution over environment variables. We prove that OFFER converges to a locally optimal policy and show experimentally that it learns better and faster than a policy gradient baseline.
Introduction
In Reinforcement Learning (RL), simulators are indispensable for training agents safely and cost-effectively. However, standard simulation-based training often suffers from a "blind spot": significant rare events (SREs). If a catastrophic event—such as a helicopter-grounding gust of wind—occurs with only 0.1% probability, a standard RL agent may never encounter it frequently enough to learn a robust response.
OFFER (Off-Environment Reinforcement Learning), presented at AAAI 2017, addresses this by enabling agents to face these rare events more frequently during training without biasing the final policy.
Problem Formulation
Most RL algorithms assume the simulator's event distribution should mirror the real world. However, for high-stakes tasks like autonomous driving, sampling from the "true" distribution is inefficient; one might need to drive millions of miles to experience a single near-miss.
The authors identify environment variables—stochastic, uncontrollable features in the real world (e.g., wind speed, friction) that are controllable in a simulator. Standard policy gradient methods fail to exploit this control, leading to high-variance gradient estimates when rare, high-impact events occur. These "surprises" can destabilize the learning process with massive gradient updates.
The OFFER Method
OFFER introduces a dual-optimization approach:
- Primary Optimization (The Actor): Updates the agent's policy ($\theta$). To handle rewards from rare events that may be orders of magnitude larger than normal rewards, OFFER uses a stochastic variant of Newton’s method. By estimating the diagonal of the Hessian, the algorithm provides adaptive learning rates for each policy feature, preventing rare-event penalties from destabilizing training.
- Secondary Optimization (The Environment): This core innovation learns a proposal distribution ($\psi$) for environment variables. Instead of the "true" distribution, OFFER samples from a distribution that minimizes the variance of the policy gradient.
To maintain accuracy, Importance Sampling is used. Updates from artificially frequent "rare" events are down-weighted, ensuring the agent learns a policy optimized for the real-world distribution.
Experimental Evaluation
The authors evaluated OFFER on two main benchmarks: Modified Mountain Car (with unaliased, aliased, and 4-featured variants) and a Robotic Arm task. In the Mountain Car tasks, rare events significantly increased penalties, while in the Robotic Arm task, rare events involved potential collisions with a wall.
Key Results and Insights
- Superior Performance: In Mountain Car tasks, baseline policy gradient methods showed negligible learning due to the rarity of the high-penalty events. OFFER successfully learned to handle them.
- Adaptive Sampling: In the unaliased Mountain Car task, OFFER’s proposal distribution sampled the rare event ~80% of the time, despite its 1% real-world frequency.
- Safety and Robustness: In the Robotic Arm task, OFFER learned a safe policy avoiding the wall, whereas the baseline failed to incorporate the wall-hit penalty effectively.
- Convergence: The paper provides a formal proof that OFFER converges to a locally optimal policy.
Conclusion
OFFER demonstrates that "off-environment" training—learning about a target environment while training in a biased "proposal" environment—is a powerful paradigm for handling high-impact rare events. By automatically discovering these events and using second-order optimization for stability, OFFER provides a path toward more robust RL agents in complex, real-world scenarios.
Summary updated