Learning to Sample with Local and Global Contexts in Experience Replay Buffer

Youngmin Oh, Kimin Lee, Jinwoo Shin, Eunho Yang, Sung Ju Hwang

Overview

Neural Experience Replay Sampler (NERS) is a learning-based approach that optimizes how reinforcement learning (RL) agents "remember" and learn from past experiences. By using a novel neural architecture that considers both individual transition data and the broader context of the entire replay buffer, NERS selects diverse and informative samples, significantly boosting the performance and sample efficiency of state-of-the-art RL algorithms.

The Problem: Inefficient Experience Replay

In off-policy reinforcement learning, agents store their past experiences (transitions) in an Experience Replay Buffer. To learn, they periodically sample batches from this buffer to update their internal models.

The standard approach is uniform random sampling, but this is inefficient because not all experiences are equally "teachable." Rule-based methods like Prioritized Experience Replay (PER) try to fix this by focusing on transitions with high "TD-error" (experiences that were most surprising to the agent). However, PER has a major flaw: it treats every transition independently. This often leads to sampling highly biased, redundant batches—for example, the agent might keep practicing the same "surprising" mistake over and over while ignoring other diverse states necessary for a well-rounded policy.

The authors identify a need for a sampler that understands relative importance: it shouldn't just ask "Is this transition important?" but "Is this transition important given what else I am learning right now?"

The Solution: Neural Experience Replay Sampler (NERS)

The authors introduce NERS, a permutation-equivariant neural network designed to process sets of transitions and score their relative importance.

Architecture of the Neural Experience Replay Sampler (NERS)
  • Local and Global Context: NERS uses two sub-networks. The Local Network extracts features from individual transitions (state, action, reward, TD-error, Q-value). The Global Network aggregates information across a batch of transitions to understand the "big picture" of the current experience set.
  • Permutation Equivariance: Because the order of transitions in a buffer shouldn't matter, the architecture is designed so that shuffling the input transitions results in the same shuffled output scores.
  • Learning to Sample: NERS is trained using the REINFORCE algorithm. Its "reward" (the replay reward) is based on the actual performance gain of the RL agent. If a specific sampling strategy helps the agent's score go up, NERS is rewarded.
  • Efficiency: To keep the system fast, NERS uses a sum-tree data structure (similar to PER), allowing it to sample from massive buffers with $O(\log |B|)$ complexity.

Key Insights

  • Diversity is as Important as Surprisal: While TD-error (surprisal) is a good signal, NERS discovers that sampling a diverse set of transitions—even those with lower error—prevents the agent from over-fitting to specific "difficult" parts of the environment.
  • Adaptive Strategies: NERS naturally learns a curriculum. Early in training, it focuses heavily on high TD-error transitions to help the agent's "critic" (value estimator) stabilize. Later, it shifts to sampling transitions with high Q-values to help the "actor" (policy) fine-tune its behavior.
  • Feature Richness: Unlike previous learning-based samplers that only looked at rewards or errors, NERS benefits significantly from seeing the raw states and actions. This allows the sampler to recognize and avoid redundant "clusters" of experience.

Performance Highlights

NERS was evaluated across a wide range of standard RL benchmarks, including MuJoCo (continuous control), OpenAI Gym, and Atari 2600 (discrete control). It consistently outperformed standard baselines (Random, PER, and ERO) across almost all environments.

Training performance comparison
  • Continuous Control: On the Ant-v3 task using the TD3 algorithm, NERS improved the average reward by +677.81 over the best baseline.
  • Discrete Control (Atari): In a data-efficient setup (100k steps), NERS outperformed the best baseline on all 16 games tested. For example, in RoadRunner, it achieved a score of 9689.30 compared to the best baseline's 7093.13.
  • Robustness: Ablation studies confirmed that removing the "Global Context" (the ability to see the batch as a whole) led to a significant drop in performance, proving that relative importance is key to efficient sampling.

Limitations and Future Directions

  • Evaluation Overhead: Training NERS requires calculating a "replay reward," which involves periodic evaluations of the agent. While the authors proposed a variant (NERS*) that uses training episodes to mitigate this, it still adds a layer of complexity.
  • Computational Cost: While the sampling complexity is low, running the NERS neural network itself adds a small computational overhead compared to simple rule-based methods like PER.
  • Future Directions: The authors suggest exploring NERS in multi-agent RL settings or meta-learning, where the ability to curate diverse experiences is even more critical.

Summary updated

Abstract

Experience replay, which enables the agents to remember and reuse experience from the past, has played a significant role in the success of off-policy reinforcement learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metrics (e.g. TD-error). However, they may result in sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, we aim to address the issue by proposing a new learning-based sampling method that can compute the relative importance of transition. To this end, we design a novel permutation-equivariant neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) as inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS), on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due to sampling diverse and meaningful transitions by NERS that considers both local and global contexts.
BibTeX
@inproceedings{oh2021learning,
  author = {Youngmin Oh and Kimin Lee and Jinwoo Shin and Eunho Yang and Sung Ju Hwang},
  title = {{Learning to Sample with Local and Global Contexts in Experience Replay Buffer}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=gJYlaqL8i8},
}
Figure