Balancing Surprise and Stability in Experience Replay

This direction is merged into another direction: prioritized experience replay. Its last version stays readable and citable here.

Version 1 of 111 papers18 conceptsfrozen 2026-10-04

Experience replay is the cornerstone of off-policy reinforcement learning, allowing agents to break temporal correlations by revisiting past transitions. The intuitive "gold standard" for this process has long been Prioritized Experience Replay (PER), which samples transitions with high Temporal Difference (TD) error. The logic is simple: if the agent is surprised by a transition, it has more to learn from it.

However, a growing body of research suggests that this "surprise-seeking" behavior is a double-edged sword. While high-error transitions contain signal, they also introduce significant noise, bias, and instability that can cause an agent's policy to collapse.

The Overshooting Trap

The primary mechanism of instability in prioritized replay is "weight overshooting." When an agent focuses intensely on a small subset of high-error transitions, it performs aggressive gradient updates to minimize those specific errors.

Ruishan Liu and James Zou in The Effects of Memory Replay in Reinforcement Learning demonstrate that this is particularly dangerous in resource-constrained settings. In environments with small memory buffers or minibatches, prioritization causes the agent to over-correct its weights based on a narrow, unrepresentative slice of experience. This leads to oscillatory behavior where the agent "chases" high-error outliers, preventing the smooth convergence of the value function.

The Noise and Distribution Problem

A high TD error does not always signal a "learnable" moment; often, it signals stochasticity or noise. Qing Wei and colleagues argue in Deep Reinforcement Learning with Quantum-inspired Experience Replay that prioritizing transitions based solely on error causes agents to over-focus on noisy interactions. This sensitivity to outliers prevents the model from consolidating the more representative, stable patterns of the environment.

Furthermore, these "surprising" states are often far from the agent’s current policy distribution. Peiquan Sun and Wengang Zhou highlight a specific failure mode in actor-critic architectures in Attentive Experience Replay. While high-error states might be informative for the critic (to update the value function), they are often too divergent from the actor's current behavior. Forcing the policy to update on these "off-distribution" states can actively destabilize the actor, leading to policy collapse.

Catastrophic Forgetting of the Mundane

If an agent only studies for the hardest parts of an exam, it may forget the basic concepts. This phenomenon, known as catastrophic forgetting, is a major risk of aggressive prioritization. Carlos Purves and Catalina Cangea show in Goal-Conditioned Reinforcement Learning in the Presence of an Adversary that exclusively training on high-difficulty or high-error tasks degrades the agent's baseline performance.

To maintain a stable policy, the agent needs a "Goldilocks" balance: enough high-surprise transitions to drive progress, but enough "easy" or random transitions to anchor its foundational knowledge. Without this balance, the agent’s value propagation becomes unstable, a problem Diksha Goel identifies as a key reason why PER fails in complex, large-scale network environments in Unveiling the Black Box.

Beyond Error: Contextual and Causal Replay

To solve these issues, researchers are moving away from raw TD error toward more "introspective" or "structured" forms of replay:

  • Causal Context: Dheeraj Mysore Nagaraj and Ramnath Kumar propose Introspective Experience Replay, which samples the sequences leading up to a surprise. This provides the agent with the causal context of how it reached a high-error state, rather than just showing it the error in isolation.
  • State Similarity: Instead of looking for what is "surprising," Attentive Experience Replay looks for what is "relevant," prioritizing past experiences that are most similar to the agent's current state.
  • Event-Based Stratification: Thomas Walsh and Peter Stone suggest in Event Tables for Efficient Experience Replay that we should prioritize based on meaningful domain events (like reaching a goal) rather than mathematical error, creating "fast lanes" for value updates that are grounded in task success rather than prediction failure.

By shifting the focus from "what the agent doesn't know" to "what the agent needs to know to stay stable," these methods allow reinforcement learning to scale to complex, noisy, and real-world environments where simple prioritization would otherwise lead to collapse.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_balancing_surprise_and_stability_in_expe_b82fa4c0,
  title = {Balancing Surprise and Stability in Experience Replay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balancing-surprise-and-stability-in-experience-replay/rdr_b82fa4c01c88436da0ce8313f3942181},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_balancing_surprise_and_stability_in_expe_b82fa4c0_v1,
  title = {Balancing Surprise and Stability in Experience Replay},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/balancing-surprise-and-stability-in-experience-replay/rdr_b82fa4c01c88436da0ce8313f3942181/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}