Learning to Sample with Local and Global Contexts in Experience Replay Buffer

Youngmin Oh, Kimin Lee, Jinwoo Shin, Eunho Yang, Sung Ju Hwang

Abstract

Experience replay, which enables the agents to remember and reuse experience from the past, has played a significant role in the success of off-policy reinforcement learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metrics (e.g. TD-error). However, they may result in sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, we aim to address the issue by proposing a new learning-based sampling method that can compute the relative importance of transition. To this end, we design a novel permutation-equivariant neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) as inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS), on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due to sampling diverse and meaningful transitions by NERS that considers both local and global contexts.

Introduction

In off-policy Reinforcement Learning (RL), agents store past experiences—tuples of state, action, reward, and next state—in an Experience Replay (ER) buffer. Efficiently sampling from this buffer is critical for learning speed and final performance. While uniform random sampling is the standard baseline, Prioritized Experience Replay (PER) improves upon it by focusing on "surprising" transitions with high Temporal Difference (TD) error. However, PER evaluates transitions in isolation, which often leads to training batches filled with redundant or biased information. This paper introduces the Neural Experience Replay Sampler (NERS), a learned sampling policy that considers the "big picture" by incorporating both local and global contexts to determine the relative importance of transitions.

Neural Experience Replay Sampler (NERS)

Architecture

NERS employs a permutation-equivariant neural architecture to ensure that the order of transitions in the buffer does not affect the sampling priority. The architecture is composed of three primary components:

  1. Local Network ($f_l$): Extracts features from individual transitions, such as states, actions, rewards, TD-errors, and Q-values.
  2. Global Network ($f_g$): Aggregates features across a candidate set of transitions to provide a global context.
  3. Score Network ($f_s$): Integrates local and global information to produce a final priority score for each transition.
NERS Architecture: The framework extracts local features and aggregates them into a global context to calculate relative importance scores for transitions.

Training via REINFORCE

The sampler is trained using the REINFORCE algorithm. Its optimization objective, the "replay reward," is defined as the actual performance gain of the RL agent. By rewarding the sampler when its chosen transitions lead to higher agent scores, NERS directly aligns its sampling strategy with the agent's ultimate success.

Results and Analysis

NERS was validated across a wide range of benchmarks, including MuJoCo continuous control tasks and Atari discrete control games. In the Ant-v3 task, NERS achieved a mean reward of 4193.69, a substantial improvement over the best baseline (ERO) at 3515.88. In Atari, NERS outperformed the standard Rainbow agent on nearly all 16 tested games, notably improving the score in KungFuMaster by nearly 3,000 points.

Adaptive Sampling Strategy

Analysis revealed that NERS naturally adapts its strategy over time. In the early stages of training, it prioritizes transitions with high TD-errors to help the agent learn the basic physics and dynamics of the environment. As the agent matures, NERS shifts its focus toward transitions with high Q-values, refining the agent's strategy for maximizing rewards.

Visualization of different sampling strategies: (a) TD-error based, (b) Q-value based, and (c) Random sampling. NERS dynamically adapts between these strategies to optimize learning.

Conclusion and Future Work

NERS demonstrates that incorporating global context into experience replay sampling significantly enhances sample efficiency and final performance in RL. While the method introduces some computational overhead and requires frequent agent evaluation for the training signal, the gains in learning robustness are significant. Future work could explore the cross-task generalization of learned samplers, potentially allowing a NERS model trained on one set of tasks to be applied to new environments without retraining.

Summary updated

Abstract

Experience replay, which enables the agents to remember and reuse experience from the past, has played a significant role in the success of off-policy reinforcement learning (RL). To utilize the experience replay efficiently, the existing sampling methods allow selecting out more meaningful experiences by imposing priorities on them based on certain metrics (e.g. TD-error). However, they may result in sampling highly biased, redundant transitions since they compute the sampling rate for each transition independently, without consideration of its importance in relation to other transitions. In this paper, we aim to address the issue by proposing a new learning-based sampling method that can compute the relative importance of transition. To this end, we design a novel permutation-equivariant neural architecture that takes contexts from not only features of each transition (local) but also those of others (global) as inputs. We validate our framework, which we refer to as Neural Experience Replay Sampler (NERS), on multiple benchmark tasks for both continuous and discrete control tasks and show that it can significantly improve the performance of various off-policy RL methods. Further analysis confirms that the improvements of the sample efficiency indeed are due to sampling diverse and meaningful transitions by NERS that considers both local and global contexts.
BibTeX
@inproceedings{oh2021learning,
  author = {Youngmin Oh and Kimin Lee and Jinwoo Shin and Eunho Yang and Sung Ju Hwang},
  title = {{Learning to Sample with Local and Global Contexts in Experience Replay Buffer}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=gJYlaqL8i8},
}

Loading related content...

Figure