Shaped Policy Search for Evolutionary Strategies using Waypoints

Kiran Lekkala, Laurent Itti

Abstract

In this paper, we try to improve exploration in Blackbox methods, particularly Evolution strategies (ES), when applied to Reinforcement Learning (RL) problems where intermediate waypoints/subgoals are available. Since Evolutionary strategies are highly parallelizable, instead of extracting just a scalar cumulative reward, we use the state-action pairs from the trajectories obtained during rollouts/evaluations, to learn the dynamics of the agent. The learnt dynamics are then used in the optimization procedure to speed-up training. Lastly, we show how our proposed approach is universally applicable by presenting results from experiments conducted on Carla driving and UR5 robotic arm simulators.

Introduction

Evolutionary Strategies (ES) offer a compelling alternative to standard Reinforcement Learning (RL) due to their "embarrassingly parallel" nature, allowing for massive scaling across distributed systems with minimal communication overhead. However, ES often suffers from poor sample efficiency, requiring millions of trials for even relatively simple tasks. This inefficiency stems largely from the fact that standard ES treats rollouts as black boxes, ignoring the rich trajectory data and only utilizing the final scalar reward.

In "Shaped Policy Search for Evolutionary Strategies using Waypoints," the authors propose leveraging intermediate waypoints—subgoals often available in robotics and navigation—to "shape" the search process. By using trajectory data to learn an inverse dynamics model, the algorithm can guide the evolution of policy parameters toward promising regions of the search space more effectively.

The core innovation is the Shaped Policy Search technique, which transforms failed explorations into informative teaching tools. The process involves three primary steps:

  1. Learning Inverse Dynamics: During rollouts, the agent records its state and the local waypoints it attempts to reach. This data trains an Inverse Dynamics Model ($M_\phi$) that predicts the necessary action to move from a current state toward a specific waypoint.
  2. Generating Surrogate Gradients: By comparing the agent's actual actions to the "ideal" actions suggested by the inverse dynamics model (using a DAgger-like approach), the system calculates surrogate gradients.
  3. Biasing the Search via Covariance Shaping: Unlike Vanilla-ES, which typically only updates the mean of the search distribution, this method uses surrogate gradients to update the covariance matrix. This "warps" the search distribution, elongating it toward the goal and narrowing the search space to the most promising directions.
The search distribution (ellipses) is shaped and biased over successive generations to move toward the optimal parameters.

Experiments and Results

The proposed method was evaluated on two high-fidelity simulators representing distinct challenges: the CARLA Driving Simulator (navigation in a complex urban environment) and the UR5 Robotic Arm in Gazebo (following a quadratic Bezier curve trajectory).

The "Shaped" ES method was benchmarked against CMA-ES, Vanilla-ES, and Proximal Policy Optimization (PPO).

Cumulative reward comparison across CARLA (a) and UR5 Gazebo (b) tasks. The proposed method (red) demonstrates superior convergence and higher final rewards compared to baselines.

As shown in the results, the shaped policy search reached higher cumulative rewards faster than all baselines in both environments. In the CARLA simulator, it reached global optima significantly earlier in terms of wall-clock time. Furthermore, the accuracy of the inverse dynamics model improved over generations, providing increasingly reliable surrogate gradients that further accelerated training as the agent gained experience.

Conclusion

By treating dynamics as "free" information and utilizing waypoints as heuristics, Shaped Policy Search significantly improves the efficiency of Evolutionary Strategies. While the method depends on the availability of quality waypoints, it provides a robust framework for complex robotics and navigation tasks where such subgoals are naturally present. Future work may explore mitigating waypoint dependency and demonstrating Sim2Real transfer on physical hardware.

Summary updated

Abstract

In this paper, we try to improve exploration in Blackbox methods, particularly Evolution strategies (ES), when applied to Reinforcement Learning (RL) problems where intermediate waypoints/subgoals are available. Since Evolutionary strategies are highly parallelizable, instead of extracting just a scalar cumulative reward, we use the state-action pairs from the trajectories obtained during rollouts/evaluations, to learn the dynamics of the agent. The learnt dynamics are then used in the optimization procedure to speed-up training. Lastly, we show how our proposed approach is universally applicable by presenting results from experiments conducted on Carla driving and UR5 robotic arm simulators.
BibTeX
@inproceedings{lekkala2021shaped,
  author = {Kiran Lekkala and Laurent Itti},
  title = {{Shaped Policy Search for Evolutionary Strategies using Waypoints}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=ZlvLhc9YFQ},
}
Figure