The Stability Gap in Continual Reinforcement Learning

Sebastian Pusch, Matthia Sabatelli, Gido M. van de VenarXiv 2026 · arXiv

TL;DR

Researchers have identified a "stability gap" in Continual Reinforcement Learning (CRL): a sharp, temporary drop in performance on previously learned tasks immediately after a new task is introduced. Crucially, this performance dip occurs even when the agent retains full access to all past data, suggesting that the problem lies in the optimization trajectory rather than just a lack of memory.

What Problem Does This Paper Address?

In the world of Reinforcement Learning (RL), agents often struggle with "catastrophic forgetting"—the tendency to completely lose old skills when learning new ones. While the field of Continual Learning (CL) develops methods to mitigate this, evaluation usually focuses on final performance metrics.

However, final scores can hide a lot of drama. In supervised learning, researchers recently discovered a "stability gap": a transient failure where performance on Task A craters the moment Task B is introduced, only to recover later. This paper investigates whether this same phenomenon haunts Reinforcement Learning. If an agent (like a self-driving car or a robot) temporarily "forgets" how to handle a common situation every time it encounters a new one, that presents a significant safety and reliability hurdle, even if it eventually recovers.

Methods

To study this, the authors introduced a new, lightweight benchmark and tested four popular RL algorithms: DQN and SAC-D (for discrete actions) and DDPG and SAC (for continuous actions).

The core of their approach is incremental joint-training. Unlike many CL setups that limit access to old data to simulate memory constraints, the authors gave their agents full access to all tasks encountered so far. They maintained separate replay buffers for each task and sampled from them equally. This "ideal" scenario was designed to isolate whether the stability gap is a fundamental optimization issue or merely a data-scarcity issue.

To visualize the "path" the neural network takes through its parameter space, they also used linear interpolation. They compared the performance of the agent along its actual training trajectory against a straight-line path between the weights it had at the end of Task 1 and the weights it had at the end of Task 2.

Datasets

The authors proposed a new transformed-state benchmark inspired by the classic "Permuted-MNIST" from supervised learning.

Instead of changing the physics of the world, they applied different orthogonal transformations to the agent's observations for each task.

  • Environments: They used CartPole (discrete) and InvertedPendulum (continuous).
  • The Twist: For each new task, the state vector is multiplied by a random orthogonal matrix. This preserves the underlying geometry and information of the state but changes the representation entirely.
  • Task Identity: A one-hot vector is appended to the input so the agent knows which "version" of the world it is currently in.
Fig. 1: Benchmark construction and CartPole instantiation. (a) Task- specific agent inputs are constructed by applying an orthogonal transforma- tion Ti to a base state s and concatenating the result with a one-hot task iden- tity. (b) An instantiation on three CartPole tasks: each four-dimensional state is transformed by a task-specific matrix and paired with a three-dimensional task identifier, yielding a seven-dimensional input. (c) All three tasks can be learned by multitask training (left), whereas sequential fine-tuning (right) shows catastrophic forgetting. Illustrated here for SAC-D on CartPole.

Key Insights and Ideas

  • The Gap is Real: All tested algorithms showed a significant stability gap. As soon as a new task was introduced, performance on the old task plummeted before slowly climbing back up.
  • Data Isn't the Only Cure: Because the gap persists even with full access to past data (joint training), we can conclude that simply "remembering" isn't enough. The way the optimizer (like Adam) updates the weights in response to new gradients temporarily "breaks" the existing solutions for old tasks.
  • The Trajectory Matters: The linear interpolation experiments revealed an "aha moment": for some algorithms like SAC, a high-performance "bridge" exists in the parameter space between the old and new task solutions. However, the optimizer doesn't take that bridge; it takes a detour through a low-performance valley.
  • Benchmark Utility: The transformed-state benchmark provides a "sanity check" for CRL. Because the dynamics and rewards are identical across tasks, any failure to learn or retain information is purely a failure of representation and optimization.

Key Results

The researchers found that the stability gap was a consistent feature across both discrete and continuous domains.

  • Transient Drops: In Figure 2 (below), you can see the "Task 1" performance (orange line) dip sharply at the vertical dashed lines (task transitions).
  • Recovery: While the agents eventually recovered most of their performance, the "gap" lasted for tens of thousands of environment steps.
  • DQN vs. SAC: Interestingly, value-based methods like DQN seemed to struggle more with recovery compared to actor-critic methods like SAC, which often returned to near-perfect performance on Task 1 while learning Task 2.
Fig. 2: A stability gap consistently shows up in CRL. Curves show mean normalized returns across seeds for (a) DQN, (b) SAC-D, (c) DDPG, and (d) SAC. Dashed lines mark task transitions.

Limitations and Future Work

The authors acknowledge that their benchmark is intentionally simple. It only varies observation representations, leaving environment dynamics and rewards fixed. Real-world CRL often involves much more drastic changes (e.g., a robot moving from a carpeted floor to a slippery one).

The main takeaway is a call to action for the research community: we need to look beyond what we optimize (the loss function) and start looking at how we optimize (the training trajectory). Future work could involve developing new optimization techniques or "warm-up" periods that prevent the agent from falling into these performance valleys during task transitions.

Summary updated

Abstract

Continual reinforcement learning (CRL) aims to enable agents to adapt to changing tasks while preserving previously acquired capabilities. Evaluation often relies on final aggregate metrics that can hide temporary performance degradation during adaptation. This work introduces a simple CRL benchmark that varies observation representations while preserving environment dynamics and rewards, supporting proof-of-concept demonstrations, inexpensive experimentation, and easily reproducible sanity checks. Using this benchmark, we demonstrate the stability gap in CRL: a substantial drop in prior-task performance after the introduction of new tasks, followed by partial or complete recovery. This pattern occurs despite the agent having complete access to past tasks.
BibTeX
@misc{pusch2026stability,
  author = {Sebastian Pusch and Matthia Sabatelli and Gido M. van de Ven},
  title = {{The Stability Gap in Continual Reinforcement Learning}},
  year = {2026},
  eprint = {2611.26252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26252},
}
Figure