COOM: A Game Benchmark for Continual Reinforcement Learning

Tristan Tomilin, Meng Fang, Yudi Zhang, Mykola PechenizkiyNeurIPS 2023 · OpenReview

Continual Reinforcement Learning (CRL) aims to create agents capable of learning a sequence of tasks without forgetting old ones or losing the ability to learn new ones. However, the field has historically struggled with inconsistent metrics, toy 2D environments, and computationally prohibitive benchmarks. Existing benchmarks often focus merely on visual changes while ignoring shifts in the agent's actual goals. To address this, researchers introduce COOM (Continual DOOM), a computationally efficient benchmark for pixel-based CRL set in 3D environments. Built on the ViZDoom engine, COOM provides a 3D, embodied perspective while remaining lightweight enough to run on standard university budgets.

The COOM Benchmark

The benchmark consists of 8 distinct scenarios (e.g., Pitfall, Arms Dealer, Hide and Seek, Floor is Lava) and several sequence types designed to test different aspects of continual learning:

  • Cross-Domain (CD): The goal remains the same (e.g., "Run and Gun"), but visuals like textures, lighting, and enemy types change.
  • Cross-Objective (CO): Both visuals and goals change (e.g., shifting from hunting enemies to avoiding them).
  • Cross-Objective Challenge (COC): A harder version of the CO sequence with sparser rewards and complex dynamics.
  • CD16/CO16: Longer sequences repeating tasks to test long-term stability.
The 'Health Gathering' scenario in COOM
The 'Floor is Lava' scenario in COOM

Methods Evaluated

The authors evaluated three main families of Continual Learning (CL) methods using Soft Actor-Critic (SAC) as the base RL engine:

  1. Regularization-based (e.g., EWC, MAS, L2): Penalize changes to "important" weights to prevent forgetting.
  2. Structure-based (e.g., PackNet): Freeze specific parts of the neural network for specific tasks, carving out dedicated capacity for each skill.
  3. Rehearsal-based (e.g., AGEM, ClonEx-SAC): Maintain a small memory of past experiences to replay while learning new tasks.

Key Insights & Results

  • Task Interference is the Main Bottleneck: Agents handle visual distortions (CD sequences) relatively well. However, when the objective changes (CO sequences), performance drops sharply due to catastrophic forgetting.
  • Top Performers: PackNet (structure-based) achieved the highest overall Average Performance (0.74), closely followed by ClonEx-SAC (rehearsal-based, 0.73). PackNet succeeds by physically isolating task knowledge, preventing the network overlap that causes forgetting.
  • Regularization Shortfalls: Regularization methods (L2, EWC) proved too rigid. While they prevented forgetting, they severely hindered the agent's ability to learn new tasks (low plasticity).
  • Critic Regularization Hurts: A surprising technical insight is that regularizing the critic (the reward estimator) actually hurts performance. It is better to let the critic adapt freely to the current task while focusing regularization efforts on the actor.
  • The Hard Challenge Remains Unsolved: The COC sequence remains largely unsolved. Even the best method (PackNet) only achieved a success score of 0.20, highlighting a massive gap for future research. Furthermore, Forward Transfer—using past knowledge to speed up future learning—was generally low across all methods.
  • No "Loss of Plasticity" (Yet): Some recent CRL research suggests agents eventually lose the ability to learn anything new. Interestingly, the base SAC agent did not exhibit this loss on COOM, suggesting the "plasticity" problem might be highly environment-dependent.

Limitations and Future Work

The benchmark currently relies on "Task-Incremental Learning," where the agent is explicitly told when a task changes, unlike real-world scenarios where agents must infer these boundaries themselves. Additionally, the study tested a fixed task ordering and simplified action spaces to keep the benchmark learnable. Future work will likely explore "Class-Incremental" settings with unknown task boundaries and aim to develop methods that achieve positive forward transfer in complex 3D worlds.

Summary updated

Abstract

The advancement of continual reinforcement learning (RL) has been facing various obstacles, including standardized metrics and evaluation protocols, demanding computational requirements, and a lack of widely accepted standard benchmarks. In response to these challenges, we present COOM ($\textbf{C}$ontinual D$\textbf{OOM}$), a continual RL benchmark tailored for embodied pixel-based RL. COOM presents a meticulously crafted suite of task sequences set within visually distinct 3D environments, serving as a robust evaluation framework to assess crucial aspects of continual RL, such as catastrophic forgetting, knowledge transfer, and sample-efficient learning. Following an in-depth empirical evaluation of popular continual learning (CL) methods, we pinpoint their limitations, provide valuable insight into the benchmark and highlight unique algorithmic challenges. This makes our work the first to benchmark image-based CRL in 3D environments with embodied perception. The primary objective of the COOM benchmark is to offer the research community a valuable and cost-effective challenge. It seeks to deepen our comprehension of the capabilities and limitations of current and forthcoming CL methods in an RL setting. The code and environments are open-sourced and accessible on GitHub.
BibTeX
@inproceedings{tomilin2023coom,
  author = {Tristan Tomilin and Meng Fang and Yudi Zhang and Mykola Pechenizkiy},
  title = {{COOM: A Game Benchmark for Continual Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=kOfZRS8gPq},
}
Figure