Introduction
Reinforcement learning (RL) in robotics often struggles with the "needle in a haystack" problem. In environments with continuous state-action spaces and sparse rewards—where an agent only receives feedback upon successful task completion—learning can be prohibitively slow. While techniques like Hindsight Experience Replay (HER) have made strides, agents still frequently fail to explore effectively or get trapped in local minima. This research introduces HiER+, a framework designed to bridge this gap by better exploiting rare successes and systematically managing task difficulty.
The HiER+ Framework
The HiER+ framework is a two-pronged approach to curriculum learning that integrates into standard off-policy RL algorithms. It addresses both how data is collected and how it is exploited during training.
Highlight Experience Replay (HiER)
HiER is a data exploitation method that maintains a secondary, smaller replay buffer ($B_{hier}$) specifically for "highlight" episodes—those where the agent achieved a total reward exceeding a threshold $\lambda$. During training, the agent samples transitions from both the standard buffer and this highlight buffer. Essentially, the agent creates its own "expert" demonstration buffer from its own past successes, "engraving" positive behaviors into its policy much like humans remember significant events more vividly than mundane ones.
Easy2Hard Initial State Entropy (E2H-ISE)
To further accelerate learning, the authors propose E2H-ISE, a data collection curriculum. This method controls task difficulty by adjusting the starting positions of the agent. It begins by placing the agent very close to the goal (low entropy/easy) and gradually expands the starting area toward a uniform distribution (high entropy/hard) as the agent's success rate improves. This ensures the agent consistently encounters reachable rewards, preventing it from wandering aimlessly in the early stages of training.
Experimental Results
The authors validated HiER across 8 tasks in three distinct robotic benchmarks: Panda-Gym, Gymnasium-Robotics Fetch, and Gymnasium-Robotics PointMaze.
Performance Gains
In the Panda-Gym benchmark, the HiER configuration achieved an Interquartile Mean (IQM) success rate of 0.83, compared to just 0.56 for the standard baseline. Notably, in the notoriously difficult PandaSlide-v3 task, where standard baselines often fail completely, HiER+ achieved a 0.82 IQM success rate.
Robustness and Universality
Statistical analysis using performance profiles confirmed that HiER and HiER+ exhibit "stochastic dominance" over their respective baselines, meaning they are statistically likely to perform better across a wide range of success thresholds. Furthermore, while primarily tested with Soft Actor-Critic (SAC), the framework proved effective when paired with other popular off-policy algorithms like TD3 and DDPG, demonstrating its algorithmic agnosticism.
Conclusion
HiER+ offers a simple yet powerful approach to boosting off-policy RL agents in sparse-reward environments. By synergizing implicit curricula (how data is used) and explicit curricula (what data is collected), it enables agents to learn complex manipulation and navigation tasks more efficiently. Future work will investigate how HiER+ can facilitate sim-to-real transfer, helping robust policies handle the "reality gap" when moving from simulators to physical robotic platforms.
Summary updated