Abstract
In deep reinforcement learning (RL), policy gradient methods such as PPO and IMPALA are widely used for on-policy control but often suffer from high variance and policy collapse, necessitating heuristics like entropy regularization. We introduce V-MPO, an on-policy adaptation of Maximum a Posteriori Policy Optimization (MPO) that frames policy improvement as a probabilistic inference problem. V-MPO leverages a learned state-value function $V(s)$ and enforces trust-region constraints to maintain stability. Our results demonstrate that V-MPO achieves state-of-the-art performance on the Atari-57 and DMLab-30 benchmarks, outperforming standard policy gradient baselines without requiring common regularization "tricks."
Introduction
The current landscape of on-policy reinforcement learning is dominated by policy gradient methods. While effective, these methods are sensitive to hyperparameter tuning and often require entropy regularization to prevent the policy from becoming prematurely deterministic. V-MPO addresses these limitations by moving away from direct gradient estimation of expected returns. Instead, it adopts an Expectation-Maximization (EM) style approach, iteratively identifying high-performing actions and updating the policy toward this improved distribution within a constrained trust region.
Methodology
V-MPO is an approximate policy iteration algorithm that decomposes the RL problem into two distinct steps:
- Expectation (E-Step): The algorithm identifies successful actions within a batch by calculating advantages relative to a learned state-value function $V(s)$. A non-parametric target distribution is constructed by weighting these actions. A key innovation is the "top-k" strategy, where the agent only learns from the best 50% of samples in a batch, effectively filtering out noise.
- Maximization (M-Step): The neural network policy is updated to match the target distribution. Stability is maintained by enforcing a KL-divergence constraint, which acts as a trust region to prevent drastic policy shifts. For continuous control, decoupled KL constraints for the mean and variance are used to preserve exploration.
Experimental Results
We evaluated V-MPO across several challenging domains, including the Atari-57 arcade suite and the DMLab-30 3D environment suite.
Atari-57 Benchmark
V-MPO demonstrates significant performance gains over previous on-policy state-of-the-art methods. On the Atari-57 benchmark, V-MPO achieved a median human-normalized score of 155.7%, significantly outperforming the 117.6% achieved by IMPALA-PBT.
DMLab-30 and Continuous Control
In the DMLab-30 suite, V-MPO surpassed previous scores in the multi-task setting and matched the performance of the large-scale off-policy agent R2D2 while using fewer environment frames. The algorithm's stability allows it to reach higher asymptotic performance without the need for population-based tuning (PBT).
Furthermore, V-MPO successfully solved complex continuous control tasks, such as a 56-degree-of-freedom humanoid "gaps" task from pixels, which previously required pre-learned motor primitives.
Conclusion
V-MPO represents a robust alternative to standard policy gradient methods. By treating policy improvement as probabilistic inference and utilizing trust-region constraints, it achieves superior stability and performance across a wide range of discrete and continuous control tasks. Its ability to reach higher asymptotic performance without complex hyperparameter tuning makes it a powerful tool for deep RL research.
Summary updated