Planning With Uncertainty: Deep Exploration in Model-Based Reinforcement Learning

Yaniv Oren, Matthijs T. J. Spaan, Wendelin Boehmer

TL;DR

This paper introduces OP2E, a novel approach that enhances deep exploration in model-based reinforcement learning (MBRL) by incorporating epistemic uncertainty directly into planning trees, rather than relying on value function propagation. This method, when applied to the state-of-the-art MuZero algorithm, significantly improves sample efficiency and enables agents to solve challenging exploration tasks that were previously intractable.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents, particularly in complex environments, often struggle with two key challenges: sample efficiency (requiring vast amounts of data to learn) and effective exploration (finding novel and potentially rewarding states). While model-based RL (MBRL) algorithms like MuZero have achieved impressive performance, their exploration strategies, often relying on random action selection or simple uncertainty bonuses, are insufficient for "hard-exploration" problems. These problems require agents to explore deeply and strategically over multiple time steps to discover crucial information or rewards. Existing methods that propagate uncertainty through value learning can lead to instability and slow learning. This paper aims to bridge this gap by developing a more effective way to leverage epistemic uncertainty (uncertainty due to lack of knowledge) for deep, farsighted exploration within MBRL.

Methods

The core contribution of this paper is the OP2E (Online Planning to Explore) framework. It modifies the standard Monte Carlo Tree Search (MCTS) used in MBRL algorithms like MuZero in several key ways:

  • Uncertainty Propagation in Planning Trees: Instead of propagating uncertainty through the value function, OP2E incorporates epistemic uncertainty directly into the planning tree. This involves estimating the variance of predicted rewards and values along a planned trajectory. The paper derives methods to compute these variances using first-order Taylor approximations of the learned model's dynamics and reward functions.
  • Exploratory Objective in MCTS: The standard Upper Confidence Bound for Trees (UCT) exploration heuristic is modified to include an "optimistic objective." This new heuristic encourages the agent to select actions that lead to states with higher epistemic uncertainty, thereby driving exploration.
  • Stabilized Learning from Exploratory Decisions: To handle the potentially off-policy nature of exploratory trajectories, the authors propose three training modifications:
  • Alternating Episodes: Training alternates between dedicated "exploration episodes" (following an exploration policy) and "exploitation episodes" (following a reward-maximizing policy).
  • Double Planning: During exploration episodes, two planning trees are computed: one for exploration and one for exploitation. This allows for off-policy value bootstrapping using the exploitation policy's value estimates.
  • Max Targets: For exploration episodes, value and policy targets are generated using a "max operator." This selects the better of the exploratory trajectory's outcome and the agent's current exploitation estimate, ensuring that learning still benefits from high-return exploratory experiences while mitigating the negative effects of poor ones.

Datasets

The paper evaluates its approach on two standard hard-exploration benchmarks:

  • Slide: A toy environment designed to test deep exploration, characterized by a long chain of states with a reward only at the very end.
  • Mountain Car: A classic, challenging exploration task where an agent must learn to drive a car up a hill, requiring significant exploration to discover the necessary strategy.

No new datasets were introduced.

Key Insights and Ideas

  • Planning Trees are a Better Venue for Uncertainty: Propagating epistemic uncertainty directly within planning trees, rather than through the value function, offers a more stable and effective mechanism for driving deep exploration. This decouples uncertainty propagation from the potentially unstable value learning process.
  • Exploration Can Be Explicitly Optimized: By modifying the MCTS exploration heuristic, agents can be explicitly guided towards uncertain states, leading to more directed and efficient information gathering.
  • Strategic Training is Crucial for Exploratory Data: Simply collecting exploratory data isn't enough. The proposed training modifications (alternating episodes, double planning, max targets) are essential for learning effectively from these exploratory trajectories, ensuring that the agent benefits from both exploration and exploitation.
  • "Max Targets" Mitigate Negative Exploration: The "max targets" approach is a clever way to leverage the benefits of exploratory data while avoiding the pitfalls of learning from consistently poor exploratory decisions. It ensures that the agent is still guided by its best current estimates of optimal behavior.

Key Results

  • OP2E, with both ensemble-based and state-visitation-counting uncertainty estimators, was able to effectively solve both the Slide and Mountain Car environments.
  • In contrast, vanilla MuZero was unable to solve either of these tasks within the given training budget.
  • The ablation studies demonstrated that the "max targets" variant (incorporating all proposed training modifications) generally outperformed other variants, highlighting the importance of the combined approach.
  • The paper suggests that while 0-step value targets can turn an algorithm into a completely off-policy one, they can be detrimental to learning. The "max targets" approach effectively overcomes this issue.

Limitations and Future Work

  • Deterministic Reward Schemes: The "max targets" approach is noted to be most effective in environments with deterministic reward schemes. Its applicability to stochastic reward environments may require further investigation.
  • Exploration/Exploitation Ratio: The paper used a fixed 1:1 ratio for alternating exploration and exploitation episodes. Investigating optimal or dynamically changing ratios is suggested as future work.
  • Computational Cost: Double planning, while effective, doubles the planning computation cost. The paper notes that this might be acceptable given that large planning budgets don't always yield proportional benefits, but it's still a consideration.
  • Modularity: The authors emphasize that their uncertainty propagation framework is modular and can be applied to other planning-based MBRL algorithms beyond MuZero. Future work could explore its integration with different planning mechanisms and uncertainty estimators.

Summary updated

Abstract

Deep model-based reinforcement learning has shown super-human performance in many challenging domains. Low sample efficiency and limited exploration remain however as leading obstacles in the field. In this paper, we demonstrate deep exploration in model-based RL by incorporating epistemic uncertainty into planning trees, circumventing the standard approach of propagating uncertainty through value learning. We evaluate this approach with the state of the art model-based RL algorithm MuZero, and extend its training process to stabilize learning from explicitly-exploratory decisions. Our results demonstrate that planning with uncertainty is able to achieve effective deep exploration with standard uncertainty estimation mechanisms, and with it significant gains in sample efficiency.
BibTeX
@inproceedings{oren2023planning,
  author = {Yaniv Oren and Matthijs T. J. Spaan and Wendelin Boehmer},
  title = {{Planning With Uncertainty: Deep Exploration in Model-Based Reinforcement Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=PcR6Lir5mxu},
}
Figure