Abstract
Reinforcement learning (RL) algorithms like Proximal Policy Optimization (PPO) are typically handcrafted, which can limit their performance and generalizability. This paper introduces Discovered Policy Optimisation (DPO), an algorithm found through meta-learning within the theoretically grounded Mirror Learning framework. DPO outperforms PPO across various tasks, maintains provable convergence guarantees, and provides implicit entropy maximization through a simple, closed-form mathematical expression.
Introduction
Most current RL algorithms rely on human mathematical intuition and extensive trial-and-error. While effective, these handcrafted methods can be brittle and sensitive to hyperparameters. Previous attempts to meta-learn RL algorithms often resulted in "black-box" functions that lacked theoretical guarantees or failed to generalize. This research seeks a middle ground: optimizing specific components within a framework that already guarantees convergence, ensuring the discovered algorithm is both high-performing and theoretically sound.
Methods: Mirror Learning and LPO
The authors utilize the Mirror Learning framework, which defines a broad space of RL algorithms through a "drift function"—a component that penalizes the policy for changing too rapidly.
- Learnt Policy Optimisation (LPO): The drift function was parameterized as a neural network and optimized using Evolution Strategies (ES) to maximize final returns across training environments.
- Analysis: They visualized the learned LPO objective (Figure 2) and compared its structure to standard PPO. They discovered two key features: "rollback" for negative advantages and "cautious optimism" for positive ones.
- Discovered Policy Optimisation (DPO): By analyzing these insights, the researchers distilled the complex neural network into a simple, analytic formula. This allows DPO to be implemented with just a few lines of code on top of existing PPO implementations.
Key Insights: Asymmetry and Rollback
Analysis of the meta-learned LPO objective revealed several key features that distinguish it from human-designed counterparts:
- The Power of Constraints: By restricting the search to the Mirror Learning space, the authors ensured that any discovered algorithm would be "provably correct."
- Rollback for Negative Advantage: For actions with negative advantages, DPO encourages the agent to maintain a moderate frequency of these actions rather than stopping updates entirely. This acts as implicit entropy maximization, preventing premature policy collapse.
- Cautious Optimism: For high positive rewards, DPO gradually slows the update pace, protecting the agent from overreacting to potentially noisy signals.
- Asymmetry is Key: Human-designed algorithms like PPO are often symmetric. The meta-learned LPO revealed that an asymmetric approach—treating positive and negative updates differently—is significantly more effective.
Experimental Results
The algorithms were evaluated using the Brax physics engine and Minatar benchmarks.
- Superior Performance: DPO and LPO consistently outperformed PPO across the majority of Brax environments, such as Ant, Humanoid, and HalfCheetah.
- Zero-Shot Generalization: Remarkably, an algorithm meta-trained only on the Ant environment generalized perfectly to much more complex tasks like Humanoid without any hyperparameter retuning.
- Robustness: DPO demonstrated more consistent performance across different random seeds and higher exploration levels than PPO, suggesting it is less sensitive to initialization.
Conclusion
By constraining the meta-learning search space to the Mirror Learning framework, the authors discovered an algorithm that is both theoretically sound and empirically superior to industry standards. DPO's simplicity—a two-line analytic formula—makes it a practical and powerful alternative for a wide range of reinforcement learning tasks. Future work will explore meta-learning other components of the Mirror Learning framework and introducing dynamic adaptation during training.
Summary updated