HiER: Highlight Experience Replay and Easy2Hard Curriculum Learning for Boosting Off-Policy Reinforcement Learning Agents

Dániel Horváth, Jesus Bujalance Martin, Ferenc Gábor Erdos, Zoltán Istenes, Fabien Moutarde

Introduction

Reinforcement learning (RL) in robotics often struggles with the "needle in a haystack" problem. In environments with continuous state-action spaces and sparse rewards—where an agent only receives feedback upon successful task completion—learning can be prohibitively slow. While techniques like Hindsight Experience Replay (HER) have made strides, agents still frequently fail to explore effectively or get trapped in local minima. This research introduces HiER+, a framework designed to bridge this gap by better exploiting rare successes and systematically managing task difficulty.

The HiER+ Framework

The HiER+ framework is a two-pronged approach to curriculum learning that integrates into standard off-policy RL algorithms. It addresses both how data is collected and how it is exploited during training.

The HiER+ framework integrates data collection curricula (E2H-ISE) and data exploitation (HiER) into the standard off-policy RL loop.

Highlight Experience Replay (HiER)

HiER is a data exploitation method that maintains a secondary, smaller replay buffer ($B_{hier}$) specifically for "highlight" episodes—those where the agent achieved a total reward exceeding a threshold $\lambda$. During training, the agent samples transitions from both the standard buffer and this highlight buffer. Essentially, the agent creates its own "expert" demonstration buffer from its own past successes, "engraving" positive behaviors into its policy much like humans remember significant events more vividly than mundane ones.

Easy2Hard Initial State Entropy (E2H-ISE)

To further accelerate learning, the authors propose E2H-ISE, a data collection curriculum. This method controls task difficulty by adjusting the starting positions of the agent. It begins by placing the agent very close to the goal (low entropy/easy) and gradually expands the starting area toward a uniform distribution (high entropy/hard) as the agent's success rate improves. This ensures the agent consistently encounters reachable rewards, preventing it from wandering aimlessly in the early stages of training.

Experimental Results

The authors validated HiER across 8 tasks in three distinct robotic benchmarks: Panda-Gym, Gymnasium-Robotics Fetch, and Gymnasium-Robotics PointMaze.

Performance Gains

In the Panda-Gym benchmark, the HiER configuration achieved an Interquartile Mean (IQM) success rate of 0.83, compared to just 0.56 for the standard baseline. Notably, in the notoriously difficult PandaSlide-v3 task, where standard baselines often fail completely, HiER+ achieved a 0.82 IQM success rate.

Robustness and Universality

Statistical analysis using performance profiles confirmed that HiER and HiER+ exhibit "stochastic dominance" over their respective baselines, meaning they are statistically likely to perform better across a wide range of success thresholds. Furthermore, while primarily tested with Soft Actor-Critic (SAC), the framework proved effective when paired with other popular off-policy algorithms like TD3 and DDPG, demonstrating its algorithmic agnosticism.

Conclusion

HiER+ offers a simple yet powerful approach to boosting off-policy RL agents in sparse-reward environments. By synergizing implicit curricula (how data is used) and explicit curricula (what data is collected), it enables agents to learn complex manipulation and navigation tasks more efficiently. Future work will investigate how HiER+ can facilitate sim-to-real transfer, helping robust policies handle the "reality gap" when moving from simulators to physical robotic platforms.

Summary updated

BibTeX
@article{horvath2024hier,
  author = {Dániel Horváth and Jesus Bujalance Martin and Ferenc Gábor Erdos and Zoltán Istenes and Fabien Moutarde},
  title = {{HiER: Highlight Experience Replay and Easy2Hard Curriculum Learning for Boosting Off-Policy Reinforcement Learning Agents}},
  journal = {IEEE Access 2024},
  year = {2024},
  doi = {10.1109/access.2024.3427012},
}
Figure