Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning

Jinyi Liu, Yi Ma, Jianye Hao, Yujing Hu, Yan Zheng, Tangjie Lv, Changjie Fan

Abstract

In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL. In summary, our research emphasizes the significance of trajectory-based data sampling techniques in enhancing the efficiency and performance of offline RL algorithms.

Introduction

Offline Reinforcement Learning (RL) aims to derive optimal policies from fixed datasets without further environment interaction. While online RL has benefited immensely from advanced data sampling techniques like Prioritized Experience Replay (PER), these methods often fail to translate to the offline setting. The primary reason is that standard transition-level sampling ($s, a, r, s'$) ignores the temporal structure of the data. By "shattering" trajectories into independent transitions, the agent loses the connectivity required to propagate reward signals effectively, especially in sparse-reward environments where the path to success is rare.

Methods: Prioritized Trajectory Replay

To address these limitations, we introduce Prioritized Trajectory Replay (PTR), a memory module designed to preserve and leverage the sequential nature of offline data.

Trajectory Replay (TR) and Backward Updates

Instead of individual transitions, TR stores and samples entire trajectories. During training, these trajectories are replayed in backward order. This ensures that when the value of a state at time $t$ is updated, it utilizes an already-updated, more accurate estimate of the value at time $t+1$. This "backward advantage" significantly accelerates the propagation of reward signals from the end of an episode back to the starting states.

The PTR sampling and batching process.

Weighted Backward Update

To mitigate extrapolation error—where the agent overestimates the value of actions not present in the dataset—we employ a weighted target. This target balances a standard RL objective with a SARSA-style objective that restricts updates to actions actually performed in the trajectory, ensuring the agent stays within the data distribution.

Prioritization Metrics

PTR further optimizes learning by prioritizing trajectories based on two key criteria:

  1. Trajectory Quality: Using metrics like average reward or Upper Quartile Mean (UQM) to focus on successful sequences, which is vital for sparse-reward tasks.
  2. Trajectory Uncertainty: Prioritizing sequences with low disagreement between ensemble Q-networks to ensure stable learning in dense-reward environments.

Key Insights: The Advantage of Trajectory-Level Sampling

The core benefit of PTR lies in its ability to maintain the "story" of the data. Transition-level sampling often misses the critical "bridge" states that connect early actions to eventual rewards. By replaying trajectories backward, PTR ensures that value estimates at the starting state ($s_0$) converge much faster and more reliably.

Comparison of state-level vs. trajectory-level sampling and its impact on Q-value convergence in dense and sparse reward settings.

As shown in the figures above, trajectory-based sampling (prio_traj, uni_traj) leads to significantly faster and more stable Q-value convergence at the initial state $s_0$ compared to state-level sampling (prio_state, uni_state), particularly in sparse-reward scenarios where the signal is hardest to find.

Results and Evaluation

We evaluated PTR on the D4RL benchmark across Gym Mujoco (dense rewards), Adroit (sparse rewards), and AntMaze (sparse rewards/long-term planning).

  • Massive Gains in Sparse Tasks: On the antmaze-umaze-v0 task, adding TR to the TD3+BC algorithm boosted the normalized score from 32.00 to 91.53.
  • Consistency Across Algorithms: PTR is a "plug-and-play" module that consistently improved performance when added to various baselines, including IQL and EDAC.
  • Efficiency: Despite the added complexity of managing trajectories, PTR only adds a negligible computational overhead (about 1–4 seconds per training epoch).

Conclusion

Prioritized Trajectory Replay represents a shift in how we handle data in offline RL. By moving from transition-level to trajectory-level sampling and utilizing backward updates, we can extract significantly more value from limited datasets. Our results on D4RL demonstrate that preserving temporal structure is key to solving complex, sparse-reward tasks in a data-driven manner.

Summary updated

Abstract

In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL. In summary, our research emphasizes the significance of trajectory-based data sampling techniques in enhancing the efficiency and performance of offline RL algorithms.
BibTeX
@misc{liu2023prioritized,
  author = {Jinyi Liu and Yi Ma and Jianye Hao and Yujing Hu and Yan Zheng and Tangjie Lv and Changjie Fan},
  title = {{Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning}},
  year = {2023},
  eprint = {2306.15503},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.15503},
}
Figure