Curiosity-Driven Exploration with Planning Trajectories

Tyler StreeterAAAI 2006

Introduction

In Reinforcement Learning (RL), agents must navigate the fundamental trade-off between exploitation—leveraging known information to secure rewards—and exploration—investigating unknown states to improve future decision-making. Model-based RL agents utilize "planning" by simulating future action sequences (trajectories) using an internal world model. While planning accelerates learning compared to trial-and-error, it is constrained by the accuracy of the internal model. Standard agents often succumb to "lazy" behavior, settling for local optima and failing to explore regions where larger rewards might reside. This paper proposes integrating a "curiosity drive" into the planning process to incentivize agents to actively resolve model uncertainty.

Methodology

The proposed architecture utilizes an actor-critic framework enhanced by temporal difference (TD) learning and a curiosity-driven world model. The core components include:

  1. Predictor Networks and Metaprediction: The agent employs neural networks to predict subsequent states and rewards. Crucially, it performs "metaprediction" to quantify the uncertainty of its own state-transition estimates.
  2. Uncertainty-Limited Planning: To prevent the agent from making decisions based on "hallucinations" in unexplored regions, simulations are truncated if the trajectory length exceeds a limit or if prediction uncertainty surpasses a predefined threshold.
  3. Intrinsic Curiosity Reward: The agent is granted an intrinsic reward proportional to its prediction uncertainty. This mechanism transforms "not knowing" into a valuable state, driving the agent toward areas where its internal model is weakest.

Experimental Setup: Discrete 2D Grid World

The methodology was evaluated using a custom-designed 2D Grid World. The environment is designed to test an agent's ability to bypass immediate gratification in favor of long-term gain.

The 2D Grid World environment featuring a starting point 'A', three local rewards (+0.1), and a distant global reward (+1.0).

As shown in Figure 1, the agent starts at position "A." The map contains three "small" rewards (+0.1) and one "large" reward (+1.0) located in a distant corner. A successful agent must resist the local optima of the +0.1 rewards to discover the +1.0 reward.

Results and Analysis

The performance of three agent configurations was compared over 50 experimental runs: Basic RL, RL with Planning, and RL with Planning and Curiosity.

Comparative performance of RL agents showing the significant advantage of curiosity-driven exploration.

As illustrated in Figure 2, the RL + Planning + Curiosity agent significantly outperformed the others:

  • Basic RL: Failed to explore effectively, achieving a negligible reward sum (~10).
  • RL + Planning: Quickly identified and exploited the local +0.1 rewards, reaching a plateau (Sum ~20) but failing to find the global optimum.
  • RL + Planning + Curiosity: Exhibited a distinct "exploration phase" for the first 15 trials, during which external rewards remained low as the agent prioritized reducing model uncertainty. Once the model was refined, the agent transitioned to exploitation, achieving a reward sum of approximately 80—a fourfold improvement over the non-curious planning agent.

Discussion and Limitations

The results demonstrate that curiosity acts as a force multiplier for planning. While planning allows for rapid policy updates, curiosity ensures those updates are directed toward globally optimal strategies rather than local traps.

However, several limitations remain. The current curiosity model relies on novelty and uncertainty, which is effective in deterministic environments but may fail in stochastic settings (the "noisy TV" problem), where an agent might become obsessed with unpredictable but irrelevant noise. Furthermore, while planning reduces the number of physical environment steps, it increases computational overhead per step. Future work will explore more robust curiosity mechanisms, such as Intelligent Adaptive Curiosity (IAC), to handle complex, non-deterministic real-world scenarios.

Summary updated

BibTeX
@inproceedings{streeter2006curiosity,
  author = {Tyler Streeter},
  title = {{Curiosity-Driven Exploration with Planning Trajectories}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure