Fingerprint Policy Optimisation for Robust Reinforcement Learning

Supratik Paul, Michael A. Osborne, Shimon Whiteson

Abstract

Fingerprint Policy Optimisation (FPO) introduces a novel framework designed to make reinforcement learning (RL) agents more robust by actively "stress-testing" them during training. By leveraging Bayesian Optimisation to select environment conditions that maximize policy improvement, FPO ensures agents learn to handle rare but catastrophic events that standard training methods frequently miss.

The Challenge of Significant Rare Events

In real-world deployments, robots face "environment variables"—unobservable factors like wind speed, floor friction, or sensor noise—that are randomly determined by nature. While these cannot be controlled in the wild, they can be manipulated within a simulator.

Standard RL typically samples these variables randomly during training. However, this naive approach fails to account for Significant Rare Events (SREs). These are low-probability occurrences, such as a sudden gust of wind or a slippery patch of ice, that have a massive impact on performance. Because they are rare, an agent trained via random sampling won't encounter them often enough to learn a proper response. This leads to policies that are fragile and prone to catastrophic failure when conditions deviate from the norm.

The core challenge is intelligently curating the training environment to focus on the most informative conditions without requiring a perfect model of the world's underlying physics.

Methodology: Fingerprint Policy Optimisation

To address this, the authors introduce Fingerprint Policy Optimisation (FPO). Instead of relying on random sampling, FPO uses Bayesian Optimisation (BO) to actively select the distribution of environment variables at each training iteration. The BO component aims to pick the environment settings expected to yield the largest improvement in the agent's overall performance.

Implementing this presents a major technical hurdle: Bayesian Optimisation typically requires low-dimensional inputs, but modern RL policies (parameterized by neural networks) have thousands or millions of parameters. To bridge this dimensionality gap, the authors introduce Policy Fingerprints:

  • State Fingerprint: A low-dimensional summary (an anisotropic Gaussian) of the states the agent actually visits.
  • Action Fingerprint: A summary of the distribution of actions the agent takes.

These fingerprints act as a proxy for the complex neural network. They allow the BO's Gaussian Process to understand how the current policy relates to different environment conditions.

Illustration of the Bayesian Optimization process mapping policy fingerprints to environment variables.

The training loop follows a continuous cycle: the BO selects a "hard" environment distribution based on the current fingerprint $\rightarrow$ the agent updates its policy using a standard method (like TRPO) $\rightarrow$ the new policy is evaluated $\rightarrow$ the results are fed back into the BO to refine the next stress test.

Experimental Setup

The researchers evaluated FPO on three continuous control tasks, modified to include high-impact rare events:

  1. Modified Cliff Walker: A continuous version of the classic problem where the agent must walk near a cliff. The rare event is falling off, carrying a massive penalty.
  2. Modified Half Cheetah (MuJoCo): The agent aims for a target velocity. In 2% of cases, the target velocity changes significantly, acting as an SRE.
  3. Modified Ant (MuJoCo): A quadruped robot where high speeds carry a 5% risk of "damage" (a large negative reward), requiring the agent to learn a cautious but efficient gait.

Key Insights and Results

FPO essentially creates an automated, adaptive curriculum. It doesn't just pick the universally "hardest" settings; it actively selects the settings that the current policy is most likely to learn from. By focusing on specific environment distributions, FPO introduces bias into the gradient estimates. However, the BO implicitly manages this bias to maximize the one-step improvement, leading to faster and more stable convergence than methods that try to correct for bias explicitly.

Learning curves comparing FPO variants against baselines.

Empirical results highlight several key advantages:

  • Superior Robustness: In the "Ant" environment, naive methods initially performed well but eventually collapsed because they learned to run fast while ignoring rare damage events. FPO, conversely, learned a robust policy that maintained high rewards while successfully avoiding catastrophic failure.
  • Efficiency: FPO outperformed existing "off-environment" baselines like EPOpt (which simply discards good trajectories) and OFFER (which requires a full transition model).
  • Fingerprint Versatility: Both the State and Action fingerprints performed similarly well. This suggests that as long as the fingerprint captures the essence of the policy's behavior, the specific implementation remains flexible.

Limitations and Future Work

The authors acknowledge a few areas for future development:

  • Quadrature Scaling: FPO uses numerical quadrature to estimate the expected return. While effective for the low-dimensional environment variables tested (1-2 dimensions), this may struggle to scale if the environment is defined by dozens of independent variables.
  • Discrete Spaces: The current fingerprints are designed for continuous state and action spaces. Extending this to discrete environments (like Atari games) would require a different type of fingerprinting mechanism.
  • Multi-step Lookahead: The BO currently optimizes for the next policy update. Future work could explore multi-step objectives that look further ahead into the training process to avoid local optima.

Summary updated

Abstract

Policy gradient methods ignore the potential value of adjusting environment variables: unobservable state features that are randomly determined by the environment in a physical setting, but are con...
BibTeX
@inproceedings{paul2019fingerprint,
  author = {Supratik Paul and Michael A. Osborne and Shimon Whiteson},
  title = {{Fingerprint Policy Optimisation for Robust Reinforcement Learning}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=7x5v-t4cWy},
}
Figure