Active Exploration in Dynamic Environments

Sebastian Thrun, Knut MöllerNeurIPS 1991 · OpenReview

Overview

In adaptive control and reinforcement learning, agents constantly face the "exploration-exploitation" dilemma: should they use their current knowledge to maximize rewards (exploitation) or try new things to learn more about their surroundings (exploration)?

Thrun and Möller propose a method for "active exploration" in continuous, real-valued environments. They introduce a Competence Map—a neural network trained to predict the error of the agent's own world model. By combining this map with a selective attention mechanism that switches between exploration and exploitation, the agent learns more efficiently and safely than traditional random-action approaches.

The Flaws of Random Exploration

Most connectionist (neural network) approaches at the time relied on random exploration, which suffers from three major flaws:

  1. Costly Mistakes: In the real world, random actions can lead to expensive "negative experiences," such as a robot repeatedly crashing into a wall.
  2. Inefficiency: Randomness is undirected. Agents waste time in well-mapped areas while ignoring poorly explored ones.
  3. Static Learning: Once an agent moves to a pure exploitation phase, it stops learning, rendering it unable to adapt if the environment changes.

While directed exploration existed for discrete "grid-world" problems, this paper seeks a solution for continuous, real-valued state spaces.

Core Methodology

The authors introduce a framework consisting of three main components:

  1. The World Model: A standard forward model trained via backpropagation to predict the next state given the current state and action.
  2. The Competence Map: This is the core innovation. It is a second neural network trained to estimate the accuracy (LMS error) of the World Model. It learns from actual prediction errors and uses a heuristic to "forget" accuracy in unvisited regions over time, ensuring the agent remains curious about areas it hasn't seen recently.
Architecture of the Competence Map
  1. Selective Attention ($\Gamma$): Instead of a fixed balance between exploration and exploitation, the authors use a bistable recursive function. This mechanism allows the agent's "focus of attention" to snap between two modes:
  2. Exploration Mode ($\Gamma \approx 0$): The agent seeks regions where the Competence Map predicts high error (low knowledge).
  3. Exploitation Mode ($\Gamma \approx 1$): The agent seeks to reach a goal while minimizing costs (e.g., avoiding obstacles).

The agent optimizes its actions by performing gradient descent on a combined energy function that weighs these two behaviors based on the current state of $\Gamma$.

Experimental Setup

The method was evaluated on a simulated 2D Robot Navigation Task.

2D Robot Navigation Environment
  • Environment: A 2-dimensional room containing a central obstacle and a goal position.
  • Input: The environment provides a potential field value (higher near the goal) and $x,y$ coordinates.
  • Goal: Navigate from start to goal without hitting the obstacle or walls.

Key Insights

  • Predicting Uncertainty: The breakthrough idea is that an agent can learn to predict its own ignorance. By training a network to model its own error, the agent gains a "sense" of where it needs to learn more.
  • Directed Curiosity: By minimizing "competence," the agent is effectively driven by curiosity toward the unknown.
  • Bistability over Blending: Rather than trying to do a little bit of exploration and exploitation simultaneously (which often leads to the agent getting "stuck" between conflicting goals), the selective attention mechanism ensures the agent commits to one behavior at a time.
  • Safety through Prediction: Because the agent uses a model to plan, it can "switch off" exploration if it predicts a collision, making learning significantly safer than random trials.

Results

The authors compared their Active Exploration method against random exploration and pure exploitation over 20 runs:

Metric Random Exploration Pure Exploitation Active Exploration
Crashes 9,993 11,000 4,000
Paths Found 0 1 2
Model Error ($L_2$) 2.5% 0.7% 0.4%

Active exploration was the only method that consistently found both possible paths around the obstacle. It also resulted in less than half the crashes of the other methods and produced the most accurate world model with the fewest training samples.

Limitations and Future Work

  • The "Curse of Dimensionality": The method seeks to explore the entire state-action space. While feasible for a 2D robot, this is computationally prohibitive for high-dimensional state spaces.
  • Relevance: Future work should include a mechanism to determine "relevance"—teaching the agent to ignore exploring parts of the environment that are irrelevant to its ultimate goals.
  • Dynamic Environments: While the title mentions dynamic environments, the primary test case was a static obstacle. The authors note the competence map's "forgetting" heuristic should help with moving obstacles, but this requires further empirical validation.

Summary updated

Abstract

Vhenever an agent learns to control an unknown environment, two oppos(cid:173) ing principles have to be combined, namely: exploration (long-term opti(cid:173) mization) and exploitation (short-term optimization). Many real-valued connectionist approaches to learning control realize exploration by ran(cid:173) domness in action selection. This might be disadvantageous when costs are assigned to "negative experiences" . The basic idea presented in this paper is to make an agent explore unknown regions in a more directed manner. This is achieved by a so-called competence map, which is trained to predict the controller's accuracy, and is used for guiding exploration. Based on this, a bistable system enables smoothly switching attention between two behaviors - exploration and exploitation - depending on ex(cid:173) pected costs and knowledge gain. The appropriateness of this method is demonstrated by a simple robot navigation task.
BibTeX
@inproceedings{thrun1991active,
  author = {Sebastian Thrun and Knut Möller},
  title = {{Active Exploration in Dynamic Environments}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {1991},
  url = {https://openreview.net/forum?id=H1W-WOZ_ZS},
}
Figure