Introduction
Deep Reinforcement Learning (RL) agents are notoriously susceptible to "catastrophic forgetting"—the loss of previously acquired knowledge when learning new tasks. In a typical lifelong learning scenario, an agent encounters a sequence of tasks. Using a standard First-In-First-Out (FIFO) replay buffer, new data quickly overwrites old experiences, causing the neural network to optimize for the current task at the expense of past performance. This paper introduces a selective experience replay mechanism designed to maintain a representative memory of the agent's entire history, enabling stable performance across multiple sequential tasks.
Methods: Selective Episodic Memory
The proposed architecture utilizes a dual-buffer system: a small, short-term FIFO buffer for immediate learning and a larger, long-term episodic memory. To manage the long-term memory, the authors explored several selection strategies to decide which experiences to retain:
- Surprise & Reward: Heuristics that prioritize "interesting" samples based on high Temporal Difference (TD) error or high returns.
- Global Distribution Matching: A strategy using reservoir sampling to ensure the memory reflects the overall distribution of all encountered experiences.
- Coverage Maximization: A method using KD-trees to maintain a spatially diverse set of experiences, ensuring the entire state space is covered.
Key Insights
A critical finding is that favoring "interesting" samples (Surprise or Reward) often leads to failure. While these samples can speed up initial learning, they create a biased training set that lacks the foundational "physics" of the environment needed to maintain old skills.
Global Distribution Matching emerged as the most robust strategy. By approximating an independent and identically distributed (i.i.d.) sample of the agent's history, it provides the stability required for deep learning. However, Coverage Maximization proved superior in "unbalanced" training scenarios, where it prevents frequent, "boring" tasks from drowning out rare but essential experiences.
Experimental Results
The researchers evaluated these strategies in an autonomous driving simulator (SUMO) across five sequential navigation tasks, such as turning left and crossing complex intersections.
As shown in the results, the selective replay mechanism allows the agent to maintain high success rates (often >80-90%) across all tasks simultaneously. In contrast, baseline FIFO methods saw performance on early tasks drop to near zero as soon as new tasks were introduced. Similar success was observed in a Lifelong MNIST task, where Distribution Matching was the only strategy that allowed the network to accurately classify all ten digits after sequential exposure.
Conclusion
Selective experience replay, particularly through distribution matching, offers a powerful and task-agnostic solution to catastrophic forgetting in RL. While computationally efficient, future work could improve the scalability of coverage-based methods to high-dimensional state spaces like raw pixels.
Summary updated