Efficient Learning of Locomotion Skills through the Discovery of Diverse Environmental Trajectory Generator Priors

Shikha Surana, Bryan Lim, Antoine Cully

TL;DR

This paper introduces Evolved Environmental Trajectory Generators (EETG), a novel method that uses Quality-Diversity algorithms to automatically discover a diverse set of specialized locomotion "priors" (called Trajectory Generators or TGs) for quadruped robots. This approach significantly improves learning efficiency and robustness across a wide range of challenging environments compared to using a single, manually defined prior.

What Problem Does This Paper Address?

Learning to control legged robots for robust locomotion across diverse and unstructured terrains is a significant challenge. While reinforcement learning (RL) has shown promise, it's notoriously sample-inefficient. A common strategy to improve this is by providing "priors" – essentially, pre-programmed movement patterns or goals – that guide the learning process. The "Policies Modulating Trajectory Generator" (PMTG) framework uses these priors effectively. However, the problem is that a single, generic prior often struggles with the complexity and variety of real-world environments. Manually crafting specialized priors for each specific scenario (like ascending stairs vs. descending them) is time-consuming and doesn't scale well. This paper tackles the challenge of efficiently learning locomotion skills by automatically discovering a diverse set of specialized priors that can adapt to various environmental conditions.

Methods

The core of the EETG method lies in combining two key components:

  1. Quality-Diversity (QD) for Trajectory Generator (TG) Discovery: EETG employs a QD algorithm, specifically a modified version of MAP-Elites, to discover a diverse set of "priors." Instead of optimizing for a single objective, QD algorithms aim to find a broad range of solutions that perform well across different "descriptors." In EETG, the "solutions" are the parameters of the Trajectory Generators (TGs), which define smooth, periodic foot trajectories. Crucially, the "descriptors" are not the behaviors of the TGs themselves, but rather the environments in which they are evaluated. This means the QD algorithm actively searches for TGs that are specialized for different environmental types (slopes, stairs, rough terrain, balance beams) and their variations. The TG itself is parameterized by 5 variables controlling swing, turn, lift, y-position, and phase offsets for each leg.
  2. Policies Modulating Trajectory Generator (PMTG) for Robust Control: Once a diverse set of specialized TGs is discovered, EETG uses the PMTG architecture. This involves training a single residual policy (a neural network) that learns to modulate the outputs of the selected TG. This policy receives the robot's state and the parameters of the chosen TG as input and outputs adjustments to the TG's trajectory. This allows the robot to adapt its learned locomotion skills to the specific TG and, by extension, the environment it's designed for. The residual policy is trained using Augmented Random Search (ARS).

Datasets

The paper does not explicitly mention using pre-existing datasets for training. Instead, it focuses on generating its own training and evaluation data within a simulated environment.

Environments: The experiments are conducted across a grid of 80 different environments, categorized into four types:

  • Slopes (varying angles)
  • Stairs (varying step heights)
  • Uneven/Rough Terrain (varying height variations)
  • Balance Beams (varying widths)

These environments are generated procedurally within the PyBullet simulator.

Key Insights and Ideas

  • Diversity of Priors is Key: The most significant insight is that a diverse set of specialized priors is far more effective for learning robust locomotion across varied terrains than a single, generic prior. This allows the robot to leverage pre-learned movement patterns tailored to specific challenges.
  • QD for Environmental Specialization: Using Quality-Diversity to directly optimize for performance across different environments (rather than just behaviors) is a powerful way to discover these specialized priors. It naturally encourages the exploration of solutions that excel in distinct conditions.
  • Sample Efficiency Gains: Discovering specialized TGs as priors significantly reduces the amount of data (samples) needed to train a robust locomotion policy. This is a critical advantage for real-world robotics where data collection is expensive and time-consuming.
  • Single Policy for Multiple Priors: The ability to use a single residual policy that can modulate a diverse set of TGs is a testament to the flexibility of the PMTG framework and the effectiveness of the learned priors.

Key Results

  • Superior Performance: EETG consistently outperforms baseline methods (vanilla PMTG with fixed or learned single TGs, and CMAES-PMTG with learned single TGs) across all tested environment types, achieving higher median total rewards.
  • Significant Efficiency Improvement: EETG is approximately 5 times more sample-efficient than the best-performing baseline (CMAES-PMTG Independent), which uses independent TGs and policies for each environment.
  • Robustness Demonstrated: The quadruped robot trained with EETG successfully traverses a wide range of challenging environments, including slopes, stairs, rough terrain, and balance beams, as visualized in Figure 1.
  • Ablation Study: The paper's ablation study suggests that while iterative refinement of the QD and policy training loops can offer minor improvements, the core benefit comes from the diverse TG discovery itself, rather than complex interdependencies between the two stages.

Limitations and Future Work

  • Deployment Assumption: A current limitation is that the method assumes knowledge of which TG to use during deployment. The paper suggests that future work could explore methods to autonomously and efficiently select a suitable TG using Bayesian Optimization or similar techniques.
  • Blind Robot: The paper notes that the robot's poor performance on balance beams is likely due to its "blindness" in this environment, where visual information would be crucial for reacting to falls. Integrating vision into the control loop is a natural extension.
  • Auto-Curricula: While EETG learns specialized priors for predefined environments, the authors suggest that future work could explore evolving environments themselves (auto-curricula) in conjunction with learning priors, which could lead to even more robust and adaptive controllers.

Summary updated

Abstract

Data-driven learning based methods have recently been particularly successful at learning robust locomotion controllers for a variety of unstructured terrains. Prior work has shown that incorporating good locomotion priors in the form of trajectory generators (TGs) is effective at efficiently learning complex locomotion skills. However, defining a good, single TG as tasks/environments become increasingly more complex remains a challenging problem as it requires extensive tuning and risks reducing the effectiveness of the prior. In this paper, we present Evolved Environmental Trajectory Generators (EETG), a method that learns a diverse set of specialised locomotion priors using Quality-Diversity algorithms while maintaining a single policy within the Policies Modulating TG (PMTG) architecture. The results demonstrate that EETG enables a quadruped robot to successfully traverse a wide range of environments, such as slopes, stairs, rough terrain, and balance beams. Our experiments show that learning a diverse set of specialized TG priors is significantly (5 times) more efficient than using a single, fixed prior when dealing with a wide range of environments.
BibTeX
@inproceedings{surana2023efficient,
  author = {Shikha Surana and Bryan Lim and Antoine Cully},
  title = {{Efficient Learning of Locomotion Skills through the Discovery of Diverse Environmental Trajectory Generator Priors}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=sj1eqvKUzN},
}
Figure