Summary
PC-MLP is a novel model-based reinforcement learning (MBRL) framework that addresses the exploration problem by combining a policy cover—an ensemble of all previously learned policies—with a principled reward bonus. Unlike previous theoretical approaches that require computationally intractable "optimistic" planners, PC-MLP works with standard black-box planning algorithms while maintaining strong theoretical guarantees and outperforming traditional MBRL on sparse-reward benchmarks.
Introduction
Model-based reinforcement learning (MBRL) is recognized for its potential sample efficiency, as agents can leverage learned transition models to simulate outcomes and accelerate learning. However, empirical MBRL algorithms often struggle with exploration, particularly in sparse-reward environments where simple heuristics like random noise or entropy regularization prove insufficient. Conversely, theoretical frameworks that guarantee exploration frequently rely on "optimistic planning oracles," which are computationally intractable for complex nonlinear systems like deep neural networks.
PC-MLP (Model Learning and Planning with Policy Cover) aims to bridge this gap. By decoupling exploration from the planning process through a reward bonus, PC-MLP enables efficient discovery of novel states while remaining compatible with standard, highly optimized planning algorithms.
Methodology
PC-MLP operates through an iterative loop consisting of four primary components:
- Policy Cover: The agent maintains an ensemble of all policies learned in previous iterations. In each episode, data is collected by sampling uniformly from this cover. This ensures the training data remains diverse and prevents the model from "forgetting" global dynamics while focusing on local trajectories.
- Model Learning: A transition model is trained via Maximum Likelihood Estimation (MLE) on the diverse dataset provided by the policy cover.
- Exploration Bonus: To encourage the discovery of novel states, the algorithm introduces a reward bonus $b_n(s, a)$. This bonus is calculated using the inverse covariance matrix of features $\phi(s, a)$ from the policy cover. If a state-action pair has features that do not align with previously seen data, the bonus is high.
- Standard Planning: Crucially, PC-MLP utilizes a standard planning oracle (e.g., TRPO or MPPI) to maximize the augmented reward $r + b_n$ within the learned model.
The practical implementation, Deep PC-MLP, utilizes deep neural networks for the transition model and employs Random Fourier Features (RFF) or random network layers to generate the exploration bonus features.
Experimental Evaluation
The authors evaluated PC-MLP across three distinct categories:
- Sparse Reward Tasks: Modified versions of MountainCar, Acrobot, and a robotic HandEgg manipulation task.
- Dense Reward Benchmarks: Ten standard MuJoCo environments (e.g., HalfCheetah, Ant, Humanoid).
- Reward-Free Exploration: An AntMaze environment where the agent receives no external reward and must explore a U-shaped corridor.
Key Results
Sparse Reward Mastery
In sparse reward settings, PC-MLP demonstrated a significant advantage over traditional MBRL baselines. In MountainCar and Acrobot, the agent achieved optimal performance almost immediately, while baselines like MBPO and SLBO failed to solve the tasks. In the complex HandEgg task, Deep PC-MLP reached a success rate of approximately 0.5 within 200,000 timesteps, whereas competitors remained near zero.
Dense Reward Performance
In standard MuJoCo environments, PC-MLP achieved the top rank in 5 out of 10 tasks. Notably, in the highly complex Humanoid environment, it reached a score of ~1775, significantly outperforming the best previous MBRL results (~1377). The exploration bonus proved beneficial even in dense reward settings by helping the agent escape local optima that trap "greedy" MBRL methods.
Efficient Maze Coverage
In the reward-free AntMaze, PC-MLP successfully achieved full coverage of the environment. As illustrated by the trajectory plots, the agent systematically explored the U-shaped maze, whereas random exploration strategies remained confined to the vicinity of the starting point.
Key Insights and Discussion
A central innovation of PC-MLP is the Policy Cover, which ensures the learned model remains globally accurate. This stability allows the agent to plan reliably into unexplored regions. By decoupling exploration from the planning process through a reward bonus, the algorithm remains compatible with existing, highly optimized motion planners.
Furthermore, the paper provides the first polynomial sample complexity guarantees for Kernelized Nonlinear Regulators (KNRs) using a standard planning oracle. This represents a significant theoretical advancement over prior work that was either limited to linear systems or required computationally impossible oracles.
Limitations and Future Work
While PC-MLP shows strong performance, it is not without limitations. The effectiveness of the exploration bonus depends on the quality of the features $\phi$. While random features performed well in these experiments, learning task-specific features online could further improve efficiency. Additionally, the "Bonus Scale" parameter requires tuning to balance exploration and reward maximization. Finally, the algorithm remains dependent on the underlying planner's ability to find global optima; if the planner stalls in local minima of the augmented reward landscape, exploration may be hindered.
Conclusion
PC-MLP effectively bridges the gap between theoretical exploration guarantees and practical MBRL implementation. By utilizing a policy cover and a novelty-based reward bonus, it enables robust exploration in sparse-reward and reward-free environments while maintaining high performance in standard dense-reward benchmarks.
Summary updated