Hierarchical Reinforcement Learning for Quadruped Locomotion

Deepali Jain, Atil Iscen, Ken Caluwaerts

TL;DR

Researchers at Google have developed a hierarchical reinforcement learning (HRL) framework that allows quadruped robots to automatically discover complex behaviors, like steering, without needing manually designed sub-tasks. By separating high-level path planning from low-level motor control and training them end-to-end, the system achieves faster adaptation to new environments and successfully transfers from simulation to a real-world Minitaur robot.

What Problem Does This Paper Address?

Legged locomotion is notoriously difficult because it requires high-frequency coordination of multiple joints while simultaneously making high-level decisions about where to go. Traditional Reinforcement Learning (RL) often struggles with these "long-horizon" tasks.

While hierarchical architectures (separating "what to do" from "how to do it") are a natural fit, they usually require a lot of "human-in-the-loop" engineering. Engineers typically have to manually define what the low-level skills are (e.g., "walk forward," "turn left") and design specific reward functions for each. This paper aims to eliminate that manual overhead by creating a system that discovers its own low-level primitives through a latent command space.

Methods

The authors propose a two-tier hierarchical structure:

  1. High-Level Policy (HL): Operates at a lower frequency. It observes the environment (robot position and orientation) and outputs two things: a latent command (a vector in a learned abstract space) and a duration (how long the low-level should run that command).
  2. Low-Level Policy (LL): Operates at a high frequency (6ms). It observes on-board sensors (IMU data) and the latent command from the HL to produce motor positions.

Key Technical Features: * End-to-End Training: Unlike many HRL methods, both levels are trained simultaneously using Augmented Random Search (ARS), a derivative-free optimization algorithm. * PMTG Integration: The system uses "Policies Modulating Trajectory Generators" (PMTG), which provides a rhythmic, cyclic basis for walking, making it easier for the RL to learn stable gaits. * Timescale Separation: The HL can choose to "hold" a command for anywhere between 0.6s and 4.2s, allowing it to focus on navigation while the LL handles the millisecond-to-millisecond balance.

Datasets

The researchers did not use a static dataset. Instead, they used:

  • Simulation: A physics-based simulation environment using pyBullet, where the robot (a Ghost Robotics Minitaur) learned to follow various curved paths.
  • Real-World Validation: The trained policies were deployed on a physical Minitaur quadruped. A PhaseSpace motion capture system provided the high-level position data to the robot in real-time.

Key Insights and Ideas

  • Emergent Primitives: Even though the researchers never told the robot how to "turn," steering behaviors automatically emerged in the latent command space. Mapping the 2D latent space revealed a clear organization of different turning angles and velocities.
  • Variable Duration is Key: The high-level policy learned to issue longer durations for straight path segments and shorter, more frequent updates for complex curves. This mimics how humans might focus more intensely when navigating a tricky corner versus walking down a straight hallway.
  • Zero-Shot Transfer (Almost): By freezing a trained low-level policy and only re-training the high-level "navigator," the robot could adapt to entirely new paths significantly faster than a "flat" (non-hierarchical) policy starting from scratch.

Key Results

  • Superior Adaptation: On a second, unseen path, the hierarchical model with a transferred low-level policy reached peak performance much faster than a flat policy or an "expert" HRL policy with pre-defined steering.
  • Sim-to-Real Success: The authors successfully demonstrated the first application (to their knowledge) of end-to-end HRL on a real quadruped. The robot was able to follow complex, curved trajectories in a lab setting despite the "noise" and "slippage" of the real world.
  • Efficiency: Because the high-level policy runs less frequently, the architecture is well-suited for distributed processing, where high-level "thinking" could happen on a remote server while low-level "reflexes" stay on the robot's local hardware.

Limitations and Future Work

  • State Space Simplicity: The current high-level observations are limited to position and orientation. The authors suggest that incorporating complex sensors like cameras (which also operate at slower timescales) would be a natural next step.
  • Task Complexity: While path-following is a great benchmark, the authors want to test if this latent space approach can discover more "agile" primitives, such as jumping over obstacles or climbing stairs.
  • Hardware Constraints: Real-world testing was limited by the capture volume of the lab's motion capture system, requiring the researchers to "virtually" move the robot to record long trajectories.

Summary updated

Abstract

Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework that can automatically learn to decompose complex locomotion tasks. A high-level policy issues commands in the form of a latent vector and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the low-level one. We test our framework on a path-following task for a dynamic quadruped robot and we show that steering behaviors automatically emerge in the latent command space as low-level skills are needed for this task. We then show efficient adaptation of the trained policy to new tasks by transfer of the trained low-level policy. Finally, we validate the policies on a real quadruped robot. To the best of our knowledge, this is the first application of end-to-end hierarchical learning to a real robotic locomotion task.
BibTeX
@inproceedings{jain2019hierarchical,
  author = {Deepali Jain and Atil Iscen and Ken Caluwaerts},
  title = {{Hierarchical Reinforcement Learning for Quadruped Locomotion}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=HHlxB1KnDKP},
}

Loading related content...

Figure