Incremental Reinforcement Learning with Dual-Adaptive ε-Greedy Exploration

Wei Ding, Siyang Jiang, Hsi-Wen Chen, Ming-Syan Chen

Introduction: The Challenge of Evolving Environments

Standard Reinforcement Learning (RL) typically assumes a static environment where the agent's state and action spaces are fixed from the start. However, real-world systems often evolve—new sensors are added, or new mechanical capabilities are unlocked. This paper formalizes this as Incremental Reinforcement Learning (IRL).

The transition from conventional RL to Incremental RL, where state and action spaces expand over time.

In IRL, simply fine-tuning a pre-trained agent often fails because the agent becomes "stuck" in its old habits. This strong inductive bias prevents it from exploring newly available states or actions that might lead to better long-term rewards. Retraining from scratch is often too computationally expensive, necessitating a more efficient way to explore expanded "territories" while preserving prior knowledge.

Methodology: Dual-Adaptive ε-Greedy Exploration (DAE)

To address the exploration challenges of IRL, the authors propose Dual-Adaptive ε-greedy Exploration (DAE). Unlike standard ε-greedy methods that use a fixed or decaying exploration rate, DAE introduces a "smart" mechanism to decide when and what to explore.

The DAE architecture: A Meta Policy determines the exploration rate ε, while an Explorer selects the least-tried actions.

The system consists of two primary components:

  1. The Meta Policy (Ψ): This component learns a state-dependent exploration rate. It uses the Temporal Difference (TD) Error as a proxy for uncertainty. High surprise (TD-error) in a specific state signals that the agent's current knowledge is insufficient, prompting the Meta Policy to increase the exploration rate (ε) for that state.
  2. The Explorer (Φ): When the agent chooses to explore, the Explorer guides it toward "novel" actions. By tracking the Relative Frequency of actions, it encourages the agent to try the least-frequently used actions in a given state, ensuring efficient coverage of the expanded action space.

By combining these, the agent adaptively decides when to explore (Meta Policy) and what to explore (Explorer), specifically targeting the "new" parts of the expanded environment.

Datasets and Benchmarks

The paper introduces two new benchmarks to evaluate Incremental RL:

  • Expanding World: A synthetic environment where the state space dimensions increase over time. As dimensions are added, the number of possible state-action combinations grows exponentially, serving as a stress test for exploration efficiency.
  • Incremental Atari: A modified version of the Arcade Learning Environment (ALE). The agent starts with basic actions (e.g., UP, DOWN) and "unlocks" more advanced actions in stages as training progresses.

Key Results and Insights

DAE demonstrated significant improvements over eight state-of-the-art baselines, including Noisy Nets, RND, and UCB:

  • Performance Boost: DAE outperformed baselines by an average of over 80% across various tasks.
  • Scalability: In the "Expanding World" benchmark, DAE solved tasks in near-linear time relative to the number of dimensions, whereas retraining from scratch showed exponential growth in training time.
  • Overcoming Bias: In Incremental Atari, DAE achieved a 22.40% improvement over the Rainbow RL baseline. It successfully discovered superior strategies in later stages that baselines missed due to being "stuck" in early-stage behaviors.

Conclusion

Incremental Reinforcement Learning represents a critical step toward more flexible and adaptive AI. By decoupling the decision of when to explore from what to explore, DAE provides a robust framework for agents to grow alongside their environments without losing the benefits of prior knowledge. Future work may look into adapting this "least-tried" logic to continuous action spaces and environments where adding new actions changes existing transition dynamics.

Summary updated

Abstract

Reinforcement learning (RL) has achieved impressive performance in various domains. However, most RL frameworks oversimplify the problem by assuming a fixed-yet-known environment and often have difficulty being generalized to real-world scenarios. In this paper, we address a new challenge with a more realistic setting, Incremental Reinforcement Learning, where the search space of the Markov Decision Process continually expands. While previous methods usually suffer from the lack of efficiency in exploring the unseen transitions, especially with increasing search space, we present a new exploration framework named Dual-Adaptive ϵ-greedy Exploration (DAE) to address the challenge of Incremental RL. Specifically, DAE employs a Meta Policy and an Explorer to avoid redundant computation on those sufficiently learned samples. Furthermore, we release a testbed based on a synthetic environment and the Atari benchmark to validate the effectiveness of any exploration algorithms under Incremental RL. Experimental results demonstrate that the proposed framework can efficiently learn the unseen transitions in new environments, leading to notable performance improvement, i.e., an average of more than 80%, over eight baselines examined.
BibTeX
@inproceedings{ding2023incremental,
  author = {Wei Ding and Siyang Jiang and Hsi-Wen Chen and Ming-Syan Chen},
  title = {{Incremental Reinforcement Learning with Dual-Adaptive ε-Greedy Exploration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i6.25899},
}
Figure