Balancing Plasticity and Stability with Fast and Slow Successor Features

Raymond Chua, Doina Precup, Blake Richards

TL;DR

When reinforcement learning (RL) agents face environments that change gradually rather than abruptly, maintaining stability matters more than injecting plasticity. This paper introduces a framework that combines predictive representations (Successor Features) with neuro-inspired multi-timescale synaptic consolidation, allowing agents to smoothly adapt to continuous environmental drift without catastrophic forgetting.

What Problem Does This Paper Address?

Deep reinforcement learning agents excel in static environments but struggle when conditions change over time. Traditionally, continual RL research evaluates agents using abrupt task switches (e.g., suddenly changing a robot's goal or switching Atari games). To combat the resulting "catastrophic forgetting," common strategies focus on injecting plasticity—making the network flexible enough to learn new things by resetting weights or increasing learning rates.

However, the real world rarely changes in discrete jumps. Instead, it evolves through gradual, continuous drift—such as mechanical wear on a robot, changing weather conditions, or shifting seasonal terrains. This paper addresses the stability-plasticity dilemma under naturalistic, continuous non-stationarity, exploring whether agents actually need more plasticity or better stability to survive gradual environmental shifts.

Methods

The authors propose a novel integration of two powerful concepts to tackle continuous drift:

  1. Successor Features (SFs): Instead of learning monolithic Q-values, SFs decouple the environment's environmental dynamics (predicting future state feature occupancies) from the reward function. This representation reduces structural interference when dynamics shift.
  2. Synaptic Consolidation (SC): Inspired by biological models of synaptic memory (Benna & Fusi, 2016), SC implements a chain of interacting hidden variables to protect learned parameters across multiple timescales. Fast timescales allow rapid adaptation to immediate changes, while slow timescales preserve long-term knowledge.

Rather than applying consolidation to Q-values or policies (the standard approach in literature), this paper applies it directly to the parameters of the Successor Features network.

Figure 2. a: Neuro-inspired synaptic consolidation model adapted from (Benna & Fusi, 2016). The visible variable, u1, represents the synaptic efficacy v, while downstream hidden variables u2, u3, ... interact bidirectionally across timescales, with beaker capacities C1 < C2 < ..., < CK and tube widths representing flow strength g1,2 > g2,3 > ..., > gK,K+1 controlling the rate of interaction between the variables. Together, the beaker sizes (Ck) and flow strength (gk,k+1) govern the effective timescales of plasticity and stability. b: The synaptic efficacy v is replaced by the parameters of SFs, thus allowing SFs to be learned across different timescales. c: Our architectural design. See section 4 for more details on training the system.

Crucially, the authors demonstrate mathematically that this multi-timescale information is only preserved when optimizing via Stochastic Gradient Descent (SGD). Popular adaptive optimizers like Adam distort the temporal scaling because their weight updates scale inversely with gradient magnitudes, neutralizing the carefully constructed consolidation dynamics. They also introduce a cross-attention mechanism over the consolidated variables to dynamically recall memories across different timescales without waiting for information to slowly propagate through the parameter chain.

Datasets

To evaluate their method without explicit task boundaries, the authors modified two standard RL benchmarks to incorporate continuous stochastic drift (via noisy sine waves, non-periodic sines, and Ornstein-Uhlenbeck processes):

  • Slippery 3D Four Rooms: A navigation environment (built on Miniworld) where the agent receives egocentric pixel observations, and the probability of an action being randomly replaced drifts continuously over time, mimicking icy or wet floors.
  • MuJoCo Control Suite (DeepMind Control Suite): Continuous control tasks of varying complexity (Half-Cheetah, Walker, Quadruped, and Humanoid) where the embodiment's body mass is continuously perturbed during training.

Key Insights and Ideas

  • Stability is the Real Bottleneck: Under gradual environmental drift, methods designed to inject plasticity (like parameter resets or continual backpropagation) perform poorly. Protecting existing knowledge (stability) is far more critical than forcing the network to be flexible.
  • Representations are Better Consolidation Targets: Consolidating predictive representations (SFs) yields significantly better performance than consolidating raw task values (Q-values), especially as environmental non-stationarity becomes more severe.
  • Slower Timescales Provide Grounding: Ablation studies showed that using more consolidation variables (6 to 9 timescales) drastically improves learning efficiency, confirming that slow-moving memory components prevent the agent's policy from drifting into failure.
  • Dynamic Scaling Beats Raw Network Capacity: Simply increasing the parameter count of baseline models (making networks larger) does not match the performance of SF+SC. The benefit stems from the multi-timescale consolidation dynamics, not just added capacity.

Key Results

Across both navigation and continuous control domains, the proposed SF + SC method consistently outperformed both purely plastic baselines and standard stability methods like Elastic Weight Consolidation (EWC).

Figure 1. Motivating stability-plasticity tradeoffs in naturalistic, continually non-stationary RL where the environment evolves gradually, rather than abruptly. To illustrate, we show (a) the Humanoid walking forward task and (b) an example of the noisy sine function used to generate smooth changes in its mass. (c) Average episode return plot and (d) Area under the curve (AUC) show that stability-preserving methods (EWC, SC) outperform purely plastic ones (CBP, P-last), with further gains from consolidating SFs (SF+SC, purple) rather than Q-values (TD3+SC, green). Plasticity injection for TD3+P-last (yellow) was performed halfway through the training.

As seen in Figure 1, on the complex Humanoid walking task under severe mass perturbations, SF+SC achieved the highest average episode return and Area Under the Curve (AUC). In contrast, plasticity-injection methods like Continual Backprop (CBP) suffered complete performance collapse. Cross-attention analysis further revealed that while fast timescales drive immediate learning, slower timescales provide the essential stability that prevents catastrophic decay over long horizons.

Limitations and Future Work

  • Optimizer Constraints: The framework is strictly limited to SGD-based updates. Combining it with popular adaptive optimizers like Adam breaks the timescale dynamics and causes instability, which may limit its drop-in applicability for architectures that depend heavily on Adam.
  • Computational Overhead: The analytical updates for multiple consolidation variables must be computed sequentially. Because these updates sit outside the standard backpropagation computational graph, they introduce a non-parallelizable computational overhead that scales with the number of timescales used.
  • Future Directions: The authors suggest that exploring more extreme settings, such as multi-agent reinforcement learning under continuous non-stationarity, remains an open and promising direction for future research.

Summary updated

Abstract

A hallmark of intelligence is the ability to adapt in non-stationary environments, yet deep Reinforcement Learning (RL) agents often struggle in such settings. Prior studies introduce non-stationarity through abrupt shifts in features or dynamics, whereas real-world environments often evolve gradually through continual drift. This distinction has important implications for the "stability-plasticity dilemma" in RL, as abrupt task changes may demand more plasticity than naturalistic settings. To address this, we modify existing 3D Miniworld and MuJoCo environments to incorporate naturalistic, continual non-stationarity, and use them to examine how stability and adaptation affect performance under continuous environmental change. We find that methods favoring stability, such as synaptic consolidation, outperform approaches focused on plasticity, such as parameters resetting. Motivated by this result, and prior evidence that Successor Features (SFs) reduce interference, we investigate whether SFs are better consolidation targets than Q-values. Across both environments, applying neuro-inspired synaptic consolidation to SFs yields superior performance on continually changing settings. Moreover, consolidation is most effective when SFs are stabilized across multiple timescales, which capture complementary aspects of gradual environmental change. Together, these results suggest that stability is more critical in continual learning when changes are gradual, and that multi-timescale consolidation of predictive representations is an effective approach.
BibTeX
@inproceedings{chua2026balancing,
  author = {Raymond Chua and Doina Precup and Blake Richards},
  title = {{Balancing Plasticity and Stability with Fast and Slow Successor Features}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=SbOzVVh3kr},
}
Figure