Discovered Policy Optimisation

Chris Lu, Jakub Grudzien Kuba, Alistair Letcher, Luke Metz, Christian Schroeder de Witt, Jakob Nicolaus Foerster

Abstract

Reinforcement learning (RL) algorithms like Proximal Policy Optimization (PPO) are typically handcrafted, which can limit their performance and generalizability. This paper introduces Discovered Policy Optimisation (DPO), an algorithm found through meta-learning within the theoretically grounded Mirror Learning framework. DPO outperforms PPO across various tasks, maintains provable convergence guarantees, and provides implicit entropy maximization through a simple, closed-form mathematical expression.

Introduction

Most current RL algorithms rely on human mathematical intuition and extensive trial-and-error. While effective, these handcrafted methods can be brittle and sensitive to hyperparameters. Previous attempts to meta-learn RL algorithms often resulted in "black-box" functions that lacked theoretical guarantees or failed to generalize. This research seeks a middle ground: optimizing specific components within a framework that already guarantees convergence, ensuring the discovered algorithm is both high-performing and theoretically sound.

Methods: Mirror Learning and LPO

The authors utilize the Mirror Learning framework, which defines a broad space of RL algorithms through a "drift function"—a component that penalizes the policy for changing too rapidly.

  1. Learnt Policy Optimisation (LPO): The drift function was parameterized as a neural network and optimized using Evolution Strategies (ES) to maximize final returns across training environments.
  2. Analysis: They visualized the learned LPO objective (Figure 2) and compared its structure to standard PPO. They discovered two key features: "rollback" for negative advantages and "cautious optimism" for positive ones.
  3. Discovered Policy Optimisation (DPO): By analyzing these insights, the researchers distilled the complex neural network into a simple, analytic formula. This allows DPO to be implemented with just a few lines of code on top of existing PPO implementations.
The LPO objective derivative, discovered via meta-learning, shows a more complex, asymmetric structure compared to the symmetric clipping of PPO.

Key Insights: Asymmetry and Rollback

Analysis of the meta-learned LPO objective revealed several key features that distinguish it from human-designed counterparts:

  • The Power of Constraints: By restricting the search to the Mirror Learning space, the authors ensured that any discovered algorithm would be "provably correct."
  • Rollback for Negative Advantage: For actions with negative advantages, DPO encourages the agent to maintain a moderate frequency of these actions rather than stopping updates entirely. This acts as implicit entropy maximization, preventing premature policy collapse.
  • Cautious Optimism: For high positive rewards, DPO gradually slows the update pace, protecting the agent from overreacting to potentially noisy signals.
  • Asymmetry is Key: Human-designed algorithms like PPO are often symmetric. The meta-learned LPO revealed that an asymmetric approach—treating positive and negative updates differently—is significantly more effective.

Experimental Results

The algorithms were evaluated using the Brax physics engine and Minatar benchmarks.

  • Superior Performance: DPO and LPO consistently outperformed PPO across the majority of Brax environments, such as Ant, Humanoid, and HalfCheetah.
  • Zero-Shot Generalization: Remarkably, an algorithm meta-trained only on the Ant environment generalized perfectly to much more complex tasks like Humanoid without any hyperparameter retuning.
  • Robustness: DPO demonstrated more consistent performance across different random seeds and higher exploration levels than PPO, suggesting it is less sensitive to initialization.
Performance curves across various Brax environments show DPO (green) and LPO (orange) consistently beating the PPO baseline (blue).

Conclusion

By constraining the meta-learning search space to the Mirror Learning framework, the authors discovered an algorithm that is both theoretically sound and empirically superior to industry standards. DPO's simplicity—a two-line analytic formula—makes it a practical and powerful alternative for a wide range of reinforcement learning tasks. Future work will explore meta-learning other components of the Mirror Learning framework and introducing dynamic adaptation during training.

Summary updated

Abstract

Tremendous progress has been made in reinforcement learning (RL) over the past decade. Most of these advancements came through the continual development of new algorithms, which were designed using a combination of mathematical derivations, intuitions, and experimentation. Such an approach of creating algorithms manually is limited by human understanding and ingenuity. In contrast, meta-learning provides a toolkit for automatic machine learning method optimisation, potentially addressing this flaw. However, black-box approaches which attempt to discover RL algorithms with minimal prior structure have thus far not outperformed existing hand-crafted algorithms. Mirror Learning, which includes RL algorithms, such as PPO, offers a potential middle-ground starting point: while every method in this framework comes with theoretical guarantees, components that differentiate them are subject to design. In this paper we explore the Mirror Learning space by meta-learning a “drift” function. We refer to the immediate result as Learnt Policy Optimisation (LPO). By analysing LPO we gain original insights into policy optimisation which we use to formulate a novel, closed-form RL algorithm, Discovered Policy Optimisation (DPO). Our experiments in Brax environments confirm state-of-the-art performance of LPO and DPO, as well as their transfer to unseen settings.
BibTeX
@inproceedings{lu2022discovered,
  author = {Chris Lu and Jakub Grudzien Kuba and Alistair Letcher and Luke Metz and Christian Schroeder de Witt and Jakob Nicolaus Foerster},
  title = {{Discovered Policy Optimisation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=bVVIZjQ2AA},
}
Figure