Overview
Neural Experience Replay Sampler (NERS) is a learning-based approach that optimizes how reinforcement learning (RL) agents "remember" and learn from past experiences. By using a novel neural architecture that considers both individual transition data and the broader context of the entire replay buffer, NERS selects diverse and informative samples, significantly boosting the performance and sample efficiency of state-of-the-art RL algorithms.
The Problem: Inefficient Experience Replay
In off-policy reinforcement learning, agents store their past experiences (transitions) in an Experience Replay Buffer. To learn, they periodically sample batches from this buffer to update their internal models.
The standard approach is uniform random sampling, but this is inefficient because not all experiences are equally "teachable." Rule-based methods like Prioritized Experience Replay (PER) try to fix this by focusing on transitions with high "TD-error" (experiences that were most surprising to the agent). However, PER has a major flaw: it treats every transition independently. This often leads to sampling highly biased, redundant batches—for example, the agent might keep practicing the same "surprising" mistake over and over while ignoring other diverse states necessary for a well-rounded policy.
The authors identify a need for a sampler that understands relative importance: it shouldn't just ask "Is this transition important?" but "Is this transition important given what else I am learning right now?"
The Solution: Neural Experience Replay Sampler (NERS)
The authors introduce NERS, a permutation-equivariant neural network designed to process sets of transitions and score their relative importance.
- Local and Global Context: NERS uses two sub-networks. The Local Network extracts features from individual transitions (state, action, reward, TD-error, Q-value). The Global Network aggregates information across a batch of transitions to understand the "big picture" of the current experience set.
- Permutation Equivariance: Because the order of transitions in a buffer shouldn't matter, the architecture is designed so that shuffling the input transitions results in the same shuffled output scores.
- Learning to Sample: NERS is trained using the REINFORCE algorithm. Its "reward" (the replay reward) is based on the actual performance gain of the RL agent. If a specific sampling strategy helps the agent's score go up, NERS is rewarded.
- Efficiency: To keep the system fast, NERS uses a sum-tree data structure (similar to PER), allowing it to sample from massive buffers with $O(\log |B|)$ complexity.
Key Insights
- Diversity is as Important as Surprisal: While TD-error (surprisal) is a good signal, NERS discovers that sampling a diverse set of transitions—even those with lower error—prevents the agent from over-fitting to specific "difficult" parts of the environment.
- Adaptive Strategies: NERS naturally learns a curriculum. Early in training, it focuses heavily on high TD-error transitions to help the agent's "critic" (value estimator) stabilize. Later, it shifts to sampling transitions with high Q-values to help the "actor" (policy) fine-tune its behavior.
- Feature Richness: Unlike previous learning-based samplers that only looked at rewards or errors, NERS benefits significantly from seeing the raw states and actions. This allows the sampler to recognize and avoid redundant "clusters" of experience.
Performance Highlights
NERS was evaluated across a wide range of standard RL benchmarks, including MuJoCo (continuous control), OpenAI Gym, and Atari 2600 (discrete control). It consistently outperformed standard baselines (Random, PER, and ERO) across almost all environments.
- Continuous Control: On the Ant-v3 task using the TD3 algorithm, NERS improved the average reward by +677.81 over the best baseline.
- Discrete Control (Atari): In a data-efficient setup (100k steps), NERS outperformed the best baseline on all 16 games tested. For example, in RoadRunner, it achieved a score of 9689.30 compared to the best baseline's 7093.13.
- Robustness: Ablation studies confirmed that removing the "Global Context" (the ability to see the batch as a whole) led to a significant drop in performance, proving that relative importance is key to efficient sampling.
Limitations and Future Directions
- Evaluation Overhead: Training NERS requires calculating a "replay reward," which involves periodic evaluations of the agent. While the authors proposed a variant (NERS*) that uses training episodes to mitigate this, it still adds a layer of complexity.
- Computational Cost: While the sampling complexity is low, running the NERS neural network itself adds a small computational overhead compared to simple rule-based methods like PER.
- Future Directions: The authors suggest exploring NERS in multi-agent RL settings or meta-learning, where the ability to curate diverse experiences is even more critical.
Summary updated