Continual Reinforcement Learning (CRL) aims to create agents capable of learning a sequence of tasks without forgetting old ones or losing the ability to learn new ones. However, the field has historically struggled with inconsistent metrics, toy 2D environments, and computationally prohibitive benchmarks. Existing benchmarks often focus merely on visual changes while ignoring shifts in the agent's actual goals. To address this, researchers introduce COOM (Continual DOOM), a computationally efficient benchmark for pixel-based CRL set in 3D environments. Built on the ViZDoom engine, COOM provides a 3D, embodied perspective while remaining lightweight enough to run on standard university budgets.
The COOM Benchmark
The benchmark consists of 8 distinct scenarios (e.g., Pitfall, Arms Dealer, Hide and Seek, Floor is Lava) and several sequence types designed to test different aspects of continual learning:
- Cross-Domain (CD): The goal remains the same (e.g., "Run and Gun"), but visuals like textures, lighting, and enemy types change.
- Cross-Objective (CO): Both visuals and goals change (e.g., shifting from hunting enemies to avoiding them).
- Cross-Objective Challenge (COC): A harder version of the CO sequence with sparser rewards and complex dynamics.
- CD16/CO16: Longer sequences repeating tasks to test long-term stability.
Methods Evaluated
The authors evaluated three main families of Continual Learning (CL) methods using Soft Actor-Critic (SAC) as the base RL engine:
- Regularization-based (e.g., EWC, MAS, L2): Penalize changes to "important" weights to prevent forgetting.
- Structure-based (e.g., PackNet): Freeze specific parts of the neural network for specific tasks, carving out dedicated capacity for each skill.
- Rehearsal-based (e.g., AGEM, ClonEx-SAC): Maintain a small memory of past experiences to replay while learning new tasks.
Key Insights & Results
- Task Interference is the Main Bottleneck: Agents handle visual distortions (CD sequences) relatively well. However, when the objective changes (CO sequences), performance drops sharply due to catastrophic forgetting.
- Top Performers: PackNet (structure-based) achieved the highest overall Average Performance (0.74), closely followed by ClonEx-SAC (rehearsal-based, 0.73). PackNet succeeds by physically isolating task knowledge, preventing the network overlap that causes forgetting.
- Regularization Shortfalls: Regularization methods (L2, EWC) proved too rigid. While they prevented forgetting, they severely hindered the agent's ability to learn new tasks (low plasticity).
- Critic Regularization Hurts: A surprising technical insight is that regularizing the critic (the reward estimator) actually hurts performance. It is better to let the critic adapt freely to the current task while focusing regularization efforts on the actor.
- The Hard Challenge Remains Unsolved: The COC sequence remains largely unsolved. Even the best method (PackNet) only achieved a success score of 0.20, highlighting a massive gap for future research. Furthermore, Forward Transfer—using past knowledge to speed up future learning—was generally low across all methods.
- No "Loss of Plasticity" (Yet): Some recent CRL research suggests agents eventually lose the ability to learn anything new. Interestingly, the base SAC agent did not exhibit this loss on COOM, suggesting the "plasticity" problem might be highly environment-dependent.
Limitations and Future Work
The benchmark currently relies on "Task-Incremental Learning," where the agent is explicitly told when a task changes, unlike real-world scenarios where agents must infer these boundaries themselves. Additionally, the study tested a fixed task ordering and simplified action spaces to keep the benchmark learnable. Future work will likely explore "Class-Incremental" settings with unknown task boundaries and aim to develop methods that achieve positive forward transfer in complex 3D worlds.
Summary updated