TL;DR
When reinforcement learning (RL) agents face environments that change gradually rather than abruptly, maintaining stability matters more than injecting plasticity. This paper introduces a framework that combines predictive representations (Successor Features) with neuro-inspired multi-timescale synaptic consolidation, allowing agents to smoothly adapt to continuous environmental drift without catastrophic forgetting.
What Problem Does This Paper Address?
Deep reinforcement learning agents excel in static environments but struggle when conditions change over time. Traditionally, continual RL research evaluates agents using abrupt task switches (e.g., suddenly changing a robot's goal or switching Atari games). To combat the resulting "catastrophic forgetting," common strategies focus on injecting plasticity—making the network flexible enough to learn new things by resetting weights or increasing learning rates.
However, the real world rarely changes in discrete jumps. Instead, it evolves through gradual, continuous drift—such as mechanical wear on a robot, changing weather conditions, or shifting seasonal terrains. This paper addresses the stability-plasticity dilemma under naturalistic, continuous non-stationarity, exploring whether agents actually need more plasticity or better stability to survive gradual environmental shifts.
Methods
The authors propose a novel integration of two powerful concepts to tackle continuous drift:
- Successor Features (SFs): Instead of learning monolithic Q-values, SFs decouple the environment's environmental dynamics (predicting future state feature occupancies) from the reward function. This representation reduces structural interference when dynamics shift.
- Synaptic Consolidation (SC): Inspired by biological models of synaptic memory (Benna & Fusi, 2016), SC implements a chain of interacting hidden variables to protect learned parameters across multiple timescales. Fast timescales allow rapid adaptation to immediate changes, while slow timescales preserve long-term knowledge.
Rather than applying consolidation to Q-values or policies (the standard approach in literature), this paper applies it directly to the parameters of the Successor Features network.
Crucially, the authors demonstrate mathematically that this multi-timescale information is only preserved when optimizing via Stochastic Gradient Descent (SGD). Popular adaptive optimizers like Adam distort the temporal scaling because their weight updates scale inversely with gradient magnitudes, neutralizing the carefully constructed consolidation dynamics. They also introduce a cross-attention mechanism over the consolidated variables to dynamically recall memories across different timescales without waiting for information to slowly propagate through the parameter chain.
Datasets
To evaluate their method without explicit task boundaries, the authors modified two standard RL benchmarks to incorporate continuous stochastic drift (via noisy sine waves, non-periodic sines, and Ornstein-Uhlenbeck processes):
- Slippery 3D Four Rooms: A navigation environment (built on Miniworld) where the agent receives egocentric pixel observations, and the probability of an action being randomly replaced drifts continuously over time, mimicking icy or wet floors.
- MuJoCo Control Suite (DeepMind Control Suite): Continuous control tasks of varying complexity (Half-Cheetah, Walker, Quadruped, and Humanoid) where the embodiment's body mass is continuously perturbed during training.
Key Insights and Ideas
- Stability is the Real Bottleneck: Under gradual environmental drift, methods designed to inject plasticity (like parameter resets or continual backpropagation) perform poorly. Protecting existing knowledge (stability) is far more critical than forcing the network to be flexible.
- Representations are Better Consolidation Targets: Consolidating predictive representations (SFs) yields significantly better performance than consolidating raw task values (Q-values), especially as environmental non-stationarity becomes more severe.
- Slower Timescales Provide Grounding: Ablation studies showed that using more consolidation variables (6 to 9 timescales) drastically improves learning efficiency, confirming that slow-moving memory components prevent the agent's policy from drifting into failure.
- Dynamic Scaling Beats Raw Network Capacity: Simply increasing the parameter count of baseline models (making networks larger) does not match the performance of SF+SC. The benefit stems from the multi-timescale consolidation dynamics, not just added capacity.
Key Results
Across both navigation and continuous control domains, the proposed SF + SC method consistently outperformed both purely plastic baselines and standard stability methods like Elastic Weight Consolidation (EWC).
As seen in Figure 1, on the complex Humanoid walking task under severe mass perturbations, SF+SC achieved the highest average episode return and Area Under the Curve (AUC). In contrast, plasticity-injection methods like Continual Backprop (CBP) suffered complete performance collapse. Cross-attention analysis further revealed that while fast timescales drive immediate learning, slower timescales provide the essential stability that prevents catastrophic decay over long horizons.
Limitations and Future Work
- Optimizer Constraints: The framework is strictly limited to SGD-based updates. Combining it with popular adaptive optimizers like Adam breaks the timescale dynamics and causes instability, which may limit its drop-in applicability for architectures that depend heavily on Adam.
- Computational Overhead: The analytical updates for multiple consolidation variables must be computed sequentially. Because these updates sit outside the standard backpropagation computational graph, they introduce a non-parallelizable computational overhead that scales with the number of timescales used.
- Future Directions: The authors suggest that exploring more extreme settings, such as multi-agent reinforcement learning under continuous non-stationarity, remains an open and promising direction for future research.
Summary updated