Introduction: The Challenge of Evolving Environments
Standard Reinforcement Learning (RL) typically assumes a static environment where the agent's state and action spaces are fixed from the start. However, real-world systems often evolve—new sensors are added, or new mechanical capabilities are unlocked. This paper formalizes this as Incremental Reinforcement Learning (IRL).
In IRL, simply fine-tuning a pre-trained agent often fails because the agent becomes "stuck" in its old habits. This strong inductive bias prevents it from exploring newly available states or actions that might lead to better long-term rewards. Retraining from scratch is often too computationally expensive, necessitating a more efficient way to explore expanded "territories" while preserving prior knowledge.
Methodology: Dual-Adaptive ε-Greedy Exploration (DAE)
To address the exploration challenges of IRL, the authors propose Dual-Adaptive ε-greedy Exploration (DAE). Unlike standard ε-greedy methods that use a fixed or decaying exploration rate, DAE introduces a "smart" mechanism to decide when and what to explore.
The system consists of two primary components:
- The Meta Policy (Ψ): This component learns a state-dependent exploration rate. It uses the Temporal Difference (TD) Error as a proxy for uncertainty. High surprise (TD-error) in a specific state signals that the agent's current knowledge is insufficient, prompting the Meta Policy to increase the exploration rate (ε) for that state.
- The Explorer (Φ): When the agent chooses to explore, the Explorer guides it toward "novel" actions. By tracking the Relative Frequency of actions, it encourages the agent to try the least-frequently used actions in a given state, ensuring efficient coverage of the expanded action space.
By combining these, the agent adaptively decides when to explore (Meta Policy) and what to explore (Explorer), specifically targeting the "new" parts of the expanded environment.
Datasets and Benchmarks
The paper introduces two new benchmarks to evaluate Incremental RL:
- Expanding World: A synthetic environment where the state space dimensions increase over time. As dimensions are added, the number of possible state-action combinations grows exponentially, serving as a stress test for exploration efficiency.
- Incremental Atari: A modified version of the Arcade Learning Environment (ALE). The agent starts with basic actions (e.g., UP, DOWN) and "unlocks" more advanced actions in stages as training progresses.
Key Results and Insights
DAE demonstrated significant improvements over eight state-of-the-art baselines, including Noisy Nets, RND, and UCB:
- Performance Boost: DAE outperformed baselines by an average of over 80% across various tasks.
- Scalability: In the "Expanding World" benchmark, DAE solved tasks in near-linear time relative to the number of dimensions, whereas retraining from scratch showed exponential growth in training time.
- Overcoming Bias: In Incremental Atari, DAE achieved a 22.40% improvement over the Rainbow RL baseline. It successfully discovered superior strategies in later stages that baselines missed due to being "stuck" in early-stage behaviors.
Conclusion
Incremental Reinforcement Learning represents a critical step toward more flexible and adaptive AI. By decoupling the decision of when to explore from what to explore, DAE provides a robust framework for agents to grow alongside their environments without losing the benefits of prior knowledge. Future work may look into adapting this "least-tried" logic to continuous action spaces and environments where adding new actions changes existing transition dynamics.
Summary updated