Dex: Incremental Learning for Complex Environments in Deep Reinforcement Learning

Nick Erickson, Qi ZhaoarXiv 2017 · OpenReview

Abstract

This paper introduces Dex, a reinforcement learning environment toolkit specialized for training and evaluation of continual learning methods as well as general reinforcement learning problems. We also present the novel continual learning method of incremental learning, where a challenging environment is solved using optimal weight initialization learned from first solving a similar easier environment. We show that incremental learning can produce vastly superior results than standard methods by providing a strong baseline method across ten Dex environments. We finally develop a saliency method for qualitative analysis of reinforcement learning, which shows the impact incremental learning has on network attention.

Introduction: The Challenge of Sparse Rewards

Deep Reinforcement Learning (RL) has achieved remarkable success in domains like Go and Atari. However, these agents typically begin training with a "blank slate" via randomly initialized weights. This approach often fails in complex environments where rewards are sparse. In games like Montezuma’s Revenge, an agent may explore for millions of steps without receiving a single reward signal, making learning from scratch practically impossible.

Furthermore, as environments grow in complexity and computational cost, the "start from scratch" paradigm becomes increasingly inefficient. The authors argue that for RL to scale, agents must adopt continual learning: the ability to leverage knowledge from simpler subtasks (such as basic movement or obstacle avoidance) to solve more difficult ones, mirroring the human learning process.

The Dex Toolkit

The primary contribution of this work is Dex, a toolkit wrapping the fast-paced geometric game Open Hexagon. Unlike standard benchmarks, Dex is specifically designed for transfer and continual learning.

  • Variety and Complexity: Dex contains hundreds of levels with varying mechanics, such as control inversion or arithmetic challenges.
  • Structured Progression: Environments are grouped into sets of related tasks, allowing researchers to study the relationship between levels.
  • Efficiency: The toolkit is designed for high-speed simulation, facilitating rapid experimentation.
A selection of environments from the Dex toolkit, showcasing the diversity of geometric challenges and mechanics.

Methods

The researchers propose a three-pronged approach:

  1. Incremental Learning: This method simplifies continual learning by training an agent sequentially through a chain of increasingly difficult, correlated environments ($E_1, E_2, \dots, E_n$). The weights learned from an easier task serve as the initialization for the next, more complex task.
  2. Modified A3C Architecture: The study utilizes the Asynchronous Advantage Actor-Critic (A3C) framework, modified to include an experience replay buffer. This "offline" adaptation ensures stability while handling the real-time nature of the Dex toolkit.
  3. Saliency Visualization: To qualitatively analyze agent behavior, the authors adapted gradient-based saliency maps to RL. These heatmaps reveal which pixels the agent "attends to" when making decisions, providing insight into how incremental learning shapes internal representations.

Key Insights and Results

The performance of a baseline agent (random initialization) was compared against an incremental agent across multiple environment sets.

The Initialization Advantage

Incremental learning acts as a powerful weight initialization technique. In environment $b_4$, the incremental agent achieved a Max Reward of 105.48, nearly triple the baseline's 35.66. Notably, the agent starting with prior knowledge performed nearly as well before training as the baseline did after a full hour of training. This suggests that the agent builds internal "feature detectors" in simpler environments that are immediately transferable to harder ones.

The Risk of Overfitting

Incremental learning is not a universal solution. In "Set $a$," the incremental agent underperformed (Max Reward of 86.57 vs. 445.85 for the baseline). The authors hypothesize that if the initial task is too simple, the agent may over-optimize for patterns that do not exist in the target environment, leading to a detrimental initialization. This highlights the critical importance of curriculum selection.

Visualizing Knowledge Retention

Saliency maps revealed that incrementally trained agents maintain attention patterns inherited from their "parent" tasks. This suggests that the network's expertise is physically preserved in the weights, allowing the agent to focus on relevant features immediately.

Saliency maps illustrating the agent's visual attention. These heatmaps show how incremental learning helps the agent focus on critical environmental features.

Limitations and Future Work

While promising, incremental learning faces several challenges:

  • Automated Curricula: Currently, task sequences are manually selected. Automating the discovery of optimal learning paths remains a major open problem.
  • Catastrophic Forgetting: The method focuses on the final task; ensuring the agent retains proficiency in earlier subtasks is a key area for future research.
  • Integration with Regularization: Future work could combine incremental learning with techniques like Elastic Weight Consolidation (EWC) to mitigate forgetting while acquiring new skills.

Summary updated

Abstract

This paper introduces Dex, a reinforcement learning environment toolkit specialized for training and evaluation of continual learning methods as well as general reinforcement learning problems. We also present the novel continual learning method of incremental learning, where a challenging environment is solved using optimal weight initialization learned from first solving a similar easier environment. We show that incremental learning can produce vastly superior results than standard methods by providing a strong baseline method across ten Dex environments. We finally develop a saliency method for qualitative analysis of reinforcement learning, which shows the impact incremental learning has on network attention.
BibTeX
@misc{erickson2017dex,
  author = {Nick Erickson and Qi Zhao},
  title = {{Dex: Incremental Learning for Complex Environments in Deep Reinforcement Learning}},
  year = {2017},
  eprint = {1706.05749},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1706.05749},
}
Figure