Scalable Experience Replay for Deep Reinforcement Learning

This direction is merged into another direction: prioritized experience replay. Its last version stays readable and citable here.

Version 1 of 120 papers20 conceptsfrozen 2026-10-04

In deep reinforcement learning, the experience replay buffer is a fundamental tool for breaking temporal correlations and improving sample efficiency. However, as agents move from simple benchmarks to complex, high-throughput environments, a "replay dilemma" emerges: the most effective ways to sample data are often the most computationally expensive.

Recent research has focused on resolving this bottleneck, moving away from naive prioritization toward methods that balance mathematical rigor with hardware-aware efficiency.

The Problem with Smart Sampling

The gold standard for experience replay is Prioritized Experience Replay (PER), which samples transitions based on their Temporal Difference (TD) error. The intuition is simple: learn more from the transitions that surprise you. However, as noted by Rui Zhao and Volker Tresp in Curiosity-Driven Experience Prioritization via Density Estimation, maintaining these priorities is a massive overhead.

Every time the agent learns, the TD errors of the sampled transitions change. In a large buffer, most priorities become "stale" almost immediately. Updating them requires constant recalculation and complex data structures like SumTrees, which Peiquan Sun and colleagues argue in Attentive Experience Replay makes them significantly slower than simple uniform sampling.

Large Batch Sub-sampling

One way to bypass the cost of global prioritization is to use a two-stage sampling process. In Large Batch Experience Replay (LaBER), Matthieu Geist and his team propose sampling a very large "surrogate" batch uniformly from the buffer first.

Instead of trying to keep the entire buffer's priorities up-to-date, LaBER only calculates the exact, current importance (like gradient norms) for this smaller subset. By selecting the final training batch from this pre-sampled pool, the agent gets the benefit of "up-to-date" prioritization without the prohibitive cost of a full backward pass for every item in the buffer. This reframes replay as a local importance sampling problem rather than a global sorting problem.

Lightweight Proxies for Learning Potential

If TD error is too expensive to track, can we find cheaper proxies for "what is worth learning?"

Researchers have proposed several "learning-free" metrics that don't require a full model update to calculate:

  • On-policyness: In Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning, Chihyeon Song and Jinkyoo Park use a metric of how well a past transition aligns with the current policy. This allows for a smooth transition from offline data to online fine-tuning without complex learned metrics.
  • Physical Energy: For robotics, Rui Zhao introduced Energy-Based Hindsight Experience Prioritization, which prioritizes episodes where the robot expends more physical energy. The intuition is that high-energy interactions (like successfully moving a heavy object) are rarer and more informative than low-energy failures.
  • Trajectory Similarity: Attentive Experience Replay prioritizes experiences that are most similar to the agent's current state, ensuring the model stays focused on "on-distribution" data that is relevant to its immediate situation.

Hardware-Centric Architectures

Even with clever algorithms, the physical location of data creates bottlenecks. Traditional RL systems store the replay buffer on the CPU, but the model lives on the GPU. This creates a constant data transfer tax.

In GEAR: A GPU-Centric Experience Replay System, Hanjing Wang and Weinan Zhang argue that for large-scale models like Transformers, the management of the buffer itself should move to the GPU. By performing trajectory selection and prioritization directly in GPU memory and using high-speed networking like RDMA, GEAR eliminates the CPU bottleneck entirely, achieving massive throughput gains.

The Importance Sampling Connection

At a theoretical level, these heuristics are all approximations of a deeper mathematical truth. Jie Tang and Pieter Abbeel showed in On a Connection between Importance Sampling and the Likelihood Ratio Policy Gradient that standard policy gradients are actually a specific, often inefficient, case of importance sampling.

By viewing the replay buffer not just as a "bucket of data" but as a way to estimate a global reward surface, agents like the Reactor (developed by Audrunas Gruslys and Marc Bellemare) can use distributional versions of algorithms like Retrace to learn more stably. The goal of modern research is to make these theoretically grounded "smart" sampling methods as fast and scalable as the "dumb" uniform ones.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scalable_experience_replay_for_deep_rein_c550a8b4,
  title = {Scalable Experience Replay for Deep Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-experience-replay-for-deep-reinforcement-learning/rdr_c550a8b4d2794cbcb55b4ceae721bdf1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_experience_replay_for_deep_rein_c550a8b4_v1,
  title = {Scalable Experience Replay for Deep Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-experience-replay-for-deep-reinforcement-learning/rdr_c550a8b4d2794cbcb55b4ceae721bdf1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}