The Effectiveness of World Models for Continual Reinforcement Learning

Samuel KesslerConference on Lifelong Learning Agents 2024 · OpenReview

TL;DR

This paper introduces "Continual-Dreamer," a novel approach that leverages world models for continual reinforcement learning (CRL). By adapting experience replay strategies and using world models for intrinsic exploration, Continual-Dreamer demonstrates superior performance, reduced forgetting, and improved transfer learning on challenging benchmarks compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Traditional reinforcement learning (RL) agents excel at mastering single tasks. However, real-world intelligent systems need to learn and adapt to a continuous stream of new tasks without forgetting previously acquired skills. This is the core challenge of Continual Reinforcement Learning (CRL). Existing CRL methods often struggle with catastrophic forgetting, require explicit task identification, or are not sample-efficient. This paper aims to bridge this gap by exploring how world models, known for their sample efficiency in single-task RL, can be effectively adapted for the more complex CRL setting.

Methods

The paper proposes "Continual-Dreamer," a framework built upon the DreamerV2 world model architecture. The key innovations lie in:

  • World Model for CRL: Utilizing a world model (specifically DreamerV2) to learn a predictive model of the environment. This model is then used to generate "imagined" experiences for training the agent's policy, making the learning process more sample-efficient.
  • Selective Experience Replay: Adapting experience replay mechanisms, a standard component in world models, to the CRL context. This involves exploring different strategies for populating and sampling from the replay buffer to mitigate forgetting. Techniques like Reservoir Sampling (RS) and Coverage Maximization (CM) are investigated for populating the buffer, while Uncertainty Sampling (US) and Reward Sampling (RWD) are explored for minibatch construction.
  • Task-Agnostic Exploration: Integrating an intrinsic exploration bonus derived from the world model's prediction uncertainty (using Plan2Explore). This allows the agent to explore new tasks without explicit task identification, a crucial feature for true continual learning.

Datasets

The proposed method is evaluated on two benchmark environments:

  • Minigrid: A suite of partially observable, sparse reward environments commonly used for RL research. The paper uses three specific tasks: DoorKey-9x9, SimpleCrossing-SN9, and LavaCrossing-9x9.
  • Minihack: A more challenging set of image-based, partially observable, and sparse reward tasks derived from the game of Nethack. The paper uses tasks from the CORA CRL benchmark, including 4-task and 8-task subsets.

Key Insights and Ideas

  • World Models as a Powerful CRL Paradigm: The research strongly suggests that world models are not only effective for single-task RL but also offer a potent and sample-efficient approach for continual learning.
  • Replay Buffer Management is Crucial: The way experience replay buffers are managed (how data is stored and sampled) is critical for balancing stability (preventing forgetting) and plasticity (learning new skills) in CRL.
  • Reservoir Sampling is a Strong Contender: Reservoir sampling emerges as a highly effective strategy for populating the replay buffer, ensuring a more uniform distribution of past experiences and significantly mitigating forgetting.
  • Task-Agnostic Exploration is Key: Leveraging the world model's uncertainty for intrinsic exploration allows for effective learning on new tasks without requiring explicit task identifiers, which is a significant advantage in many CRL scenarios.
  • Stability-Plasticity Trade-off: Increasing the replay buffer size can improve stability (reduce forgetting) but may hinder plasticity (learning new tasks quickly), highlighting a fundamental trade-off in CRL.

Key Results

  • Outperforms State-of-the-Art: Continual-Dreamer (the best configuration of DreamerV2 with reservoir sampling and Plan2Explore) outperforms state-of-the-art task-agnostic CRL methods like CLEAR on both Minigrid and Minihack benchmarks in terms of average performance.
  • Reduced Forgetting: Continual-Dreamer significantly reduces catastrophic forgetting compared to baselines, particularly on more challenging tasks.
  • Improved Forward Transfer: The method demonstrates strong forward transfer capabilities, indicating effective knowledge transfer to new tasks.
  • Sample Efficiency: By leveraging world models for imagined rollouts, Continual-Dreamer is more sample-efficient than methods like CLEAR, which require substantially more environment interactions. For instance, CLEAR needed 10x more interactions to solve simpler Minigrid tasks.
  • Task-Agnostic Success: The proposed method operates without explicit task identification, making it more generalizable to unseen tasks.

Limitations and Future Work

  • Interference: The paper acknowledges that Continual-Dreamer, while mitigating forgetting, does not explicitly prevent interference from past samples, which can still affect learning on new tasks, especially with changing reward functions or goal locations.
  • Task Data Imbalances: The selective experience replay methods explored are not specifically designed to handle highly imbalanced task data distributions, where one task might dominate the replay buffer.
  • Future Work: The authors suggest exploring continuous control CRL benchmarks and applying the proposed framework to other world model architectures. Further investigation into managing task data imbalances in replay buffers is also a promising direction.

Summary updated

Abstract

World models power some of the most efficient reinforcement learning algorithms. In this work, we showcase that they can be harnessed for continual learning – a situation when the agent faces changing environments. World models typically employ a replay buffer for training, which can be naturally extended to continual learning. We systematically study how different selective experience replay methods affect performance, forgetting, and transfer. We also provide recommendations regarding various modeling options for using world models. The best set of choices is called Continual-Dreamer, it is task-agnostic and utilizes the world model for continual exploration. Continual-Dreamer is sample efficient and outperforms state-of-the-art task-agnostic continual reinforcement learning methods on Minigrid and Minihack benchmarks.
BibTeX
@inproceedings{kessler2024effectiveness,
  author = {Samuel Kessler},
  title = {{The Effectiveness of World Models for Continual Reinforcement Learning}},
  booktitle = {Conference on Lifelong Learning Agents},
  year = {2024},
  url = {https://openreview.net/forum?id=TJoHM00qYE},
}
Figure