Hindsight Experience Replay with Kronecker Product Approximate Curvature

Dhuruva Priyan G. M, Abhik Singla, Shalabh BhatnagararXiv 2020 · arXiv

Abstract

This paper introduces TDHER+KFAC, a novel reinforcement learning framework designed for efficient training in sparse reward environments. By integrating Hindsight Experience Replay (HER) with Twin Delayed DDPG (TD3) and Kronecker-factored Approximate Curvature (K-FAC), the authors leverage natural gradients to optimize neural network policies. Experimental results on MuJoCo robotics benchmarks demonstrate that TDHER+KFAC significantly improves sample efficiency and convergence speed over standard HER implementations.

Introduction

Reinforcement Learning (RL) often struggles in environments with sparse rewards, where agents receive feedback only after completing a complex sequence of actions. Standard algorithms frequently fail to learn in such settings because they rarely encounter a "success" signal. Hindsight Experience Replay (HER) was developed to address this by replaying failed trajectories as if the agent had intended to reach the final state achieved. While HER improves sample efficiency, it is often limited by slow convergence due to first-order optimization (e.g., Adam) and Q-value overestimation issues inherent in the underlying DDPG algorithm.

Methodology: TDHER+KFAC

The authors propose a hybrid approach, TDHER+KFAC, which upgrades the optimization and architectural components of the HER framework:

  • Twin Delayed DDPG (TD3): To combat the overestimation bias of DDPG, TD3 employs two Q-functions, takes the minimum value between them for target updates, and adds noise to target actions to smooth the value landscape.
  • Kronecker-factored Approximate Curvature (K-FAC): This is the core optimization enhancement. Unlike standard Stochastic Gradient Descent (SGD), which follows the steepest descent in Euclidean space, K-FAC approximates Natural Gradient Descent. By utilizing the Fisher Information Matrix to account for the curvature of the loss function, K-FAC allows the model to take more effective steps toward the optimum.
  • Integration and Stability: K-FAC is applied to both the Actor (Policy) and Critic (Value) networks. The implementation incorporates Factored Tikhonov Damping and Momentum to stabilize updates within the HER framework.

Experimental Evaluation

The algorithm was evaluated using OpenAI’s MuJoCo Robotics environments, which are standard benchmarks for continuous control. The tasks included FetchReach, FetchPush, FetchPickAndPlace, FetchSlide, and HandReach. These environments require precise continuous control to reach goals under sparse reward conditions.

Key Results and Discussion

TDHER+KFAC consistently outperformed vanilla HER across most scenarios, particularly in terms of learning speed and final success rates.

Performance comparison of TDHER+KFAC (purple) vs. vanilla HER (cyan) across four Fetch environments: PickAndPlace, Push, Reach, and Slide.
  • Speed of Learning: In the FetchPush environment, the model began learning within 50 epochs, whereas vanilla HER required nearly 200 epochs to reach a similar stage.
  • Success Rates: For FetchPickAndPlace, the proposed method achieved a 40% success rate within 150 epochs, while vanilla HER remained below 10%.
Detailed learning curve for the FetchPickAndPlace task, highlighting the significant performance gap between TDHER+KFAC and vanilla HER.
  • Complex Tasks: On the HandReach task, the algorithm reached a 4% accuracy rate within 200 epochs, establishing a strong baseline for this difficult 15-dimensional environment.

Limitations

Despite its efficiency, K-FAC introduces computational overhead; training time per update is approximately 2.5 times longer than vanilla HER. Furthermore, the algorithm struggled with the FetchSlide task. The authors suggest that K-FAC may have difficulty differentiating between various failure trajectories in environments where complex physics (like friction and momentum) play a dominant role.

Conclusion

By accounting for the second-order geometry of the loss function, TDHER+KFAC provides a powerful solution for sparse reward tasks. The combination of HER’s data augmentation and K-FAC’s efficient natural gradient optimization creates a synergistic effect that drastically reduces the number of epochs needed for convergence in robotic control tasks.

Summary updated

Abstract

Hindsight Experience Replay (HER) is one of the efficient algorithm to solve Reinforcement Learning tasks related to sparse rewarded environments.But due to its reduced sample efficiency and slower convergence HER fails to perform effectively. Natural gradients solves these challenges by converging the model parameters better. It avoids taking bad actions that collapse the training performance. However updating parameters in neural networks requires expensive computation and thus increase in training time. Our proposed method solves the above mentioned challenges with better sample efficiency and faster convergence with increased success rate. A common failure mode for DDPG is that the learned Q-function begins to dramatically overestimate Q-values, which then leads to the policy breaking, because it exploits the errors in the Q-function. We solve this issue by including Twin Delayed Deep Deterministic Policy Gradients(TD3) in HER. TD3 learns two Q-functions instead of one and it adds noise tothe target action, to make it harder for the policy to exploit Q-function errors. The experiments are done with the help of OpenAis Mujoco environments. Results on these environments show that our algorithm (TDHER+KFAC) performs better inmost of the scenarios
BibTeX
@misc{m2020hindsight,
  author = {Dhuruva Priyan G. M and Abhik Singla and Shalabh Bhatnagar},
  title = {{Hindsight Experience Replay with Kronecker Product Approximate Curvature}},
  year = {2020},
  eprint = {2010.06142},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.06142},
}
Figure