Abstract: Legged locomotion is a challenging task for learning algorithms, especially when the task requires a diverse set of primitive behaviors. To solve these problems, we introduce a hierarchical framework to automatically decompose complex locomotion tasks. A high-level policy issues commands in a latent space and also selects for how long the low-level policy will execute the latent command. Concurrently, the low-level policy uses the latent command and only the robot's on-board sensors to control the robot's actuators. Our approach allows the high-level policy to run at a lower frequency than the low-level one. We test our framework on a path-following task for a dynamic quadruped robot and we show that steering behaviors automatically emerge in the latent command space as low-level skills are needed for this task. We then show efficient adaptation of the trained policy to a different task by transfer of the trained low-level policy. Finally, we validate the policies on a real quadruped robot. To the best of our knowledge, this is the first application of end-to-end hierarchical learning to a real robotic locomotion task.
Introduction
Legged locomotion is notoriously difficult because it requires high-frequency coordination of multiple joints while simultaneously making high-level decisions about navigation. Traditional Reinforcement Learning (RL) often struggles with these "long-horizon" tasks. While hierarchical architectures—separating "what to do" from "how to do it"—are a natural fit, they usually require significant manual engineering to define low-level skills (e.g., "walk forward," "turn left") and design specific reward functions for each. This work introduces a framework that eliminates this manual overhead by allowing a system to discover its own low-level primitives through a latent command space.
Hierarchical Architecture
The proposed system utilizes a two-tier hierarchical structure trained end-to-end using Augmented Random Search (ARS).
- High-Level Policy (HL): Operates at a lower frequency. It observes the environment (robot position and orientation) and outputs a latent command (a vector in a learned abstract space) and a duration (how long the low-level should run that command).
- Low-Level Policy (LL): Operates at a high frequency (6ms). It observes on-board sensors (IMU data) and the latent command from the HL to produce motor positions.
A key technical feature is the integration of Policies Modulating Trajectory Generators (PMTG), which provides a rhythmic, cyclic basis for walking, facilitating the learning of stable gaits. The HL can choose to "hold" a command for anywhere between 0.6s and 4.2s, allowing it to focus on navigation while the LL handles millisecond-to-millisecond balance.
Experimental Results
The framework was evaluated in a physics-based simulation (pyBullet) where a Ghost Robotics Minitaur learned to follow various curved paths.
Emergent Primitives and Adaptation
Even without explicit instructions on how to "turn," steering behaviors automatically emerged in the latent command space. Mapping this 2D space revealed a clear organization of different turning angles and velocities. Furthermore, the high-level policy learned to issue longer durations for straight segments and shorter, more frequent updates for complex curves, mimicking human-like focus during navigation.
When transferring a trained low-level policy to an entirely new path, the hierarchical model reached peak performance significantly faster than a "flat" (non-hierarchical) policy starting from scratch. This demonstrates the efficiency of reusing learned motor primitives for new high-level tasks.
Real-World Validation
The trained policies were successfully deployed on a physical Minitaur quadruped. A motion capture system provided high-level position data in real-time.
Despite the noise and slippage inherent in the real world, the robot followed complex, curved trajectories. This represents a significant milestone as the first application of end-to-end hierarchical learning to a real quadruped locomotion task.
Conclusion and Future Work
This hierarchical approach demonstrates that complex behaviors can emerge from simple high-level goals without manual sub-task design. Because the high-level policy runs less frequently, the architecture is well-suited for distributed processing, where high-level "thinking" could occur on a remote server while low-level "reflexes" remain local. Future work will explore incorporating complex sensors like cameras into the high-level state space and testing if this approach can discover more agile primitives, such as jumping or climbing.
Summary updated