Abstract
We focus on the problem of developing energy efficient controllers for quadrupedal robots. Animals can actively switch gaits at different speeds to lower their energy consumption. In this paper, we devise a hierarchical learning framework, in which distinctive locomotion gaits and natural gait transitions emerge automatically with a simple reward of energy minimization. We use evolutionary strategies (ES) to train a high-level gait policy that specifies gait patterns of each foot, while the low-level convex MPC controller optimizes the motor commands so that the robot can walk at a desired velocity using that gait pattern. We test our learning framework on a quadruped robot and demonstrate automatic gait transitions, from walking to trotting and to fly-trotting, as the robot increases its speed. We show that the learned hierarchical controller consumes much less energy across a wide range of locomotion speed than baseline controllers.
Introduction
In nature, animals exhibit remarkable energy efficiency by dynamically switching gaits—walking at low speeds, trotting at medium speeds, and galloping at high speeds. Replicating this fluidity in robotics is challenging. Traditional controllers often rely on rigid, hand-engineered gait timings that lack adaptability. Conversely, end-to-end Reinforcement Learning (RL) frequently fails to bridge the "sim-to-real" gap, where simulated successes do not translate to physical hardware due to unmodeled dynamics and safety constraints. This work introduces a framework that enables robots to learn energy-efficient gaits and natural transitions automatically, maintaining the robustness required for real-world deployment.
Hierarchical Control Architecture
The proposed system utilizes a hierarchical framework that decouples gait timing from motor execution.
- High-Level Gait Policy (RL): A neural network serves as a gait generator. Rather than commanding torques directly, it outputs five parameters: stepping frequency, swing-to-stance ratio, and phase offsets for each leg. This policy is trained via Evolutionary Strategies (CMA-ES), optimizing for a reward that balances velocity tracking with energy minimization (Cost of Transport).
- Low-Level Convex MPC: This layer receives gait parameters and the robot's state to compute optimal ground reaction forces. It manages the fundamental physics of balancing and stable foot placement using a simplified centroidal dynamics model.
By employing Evolutionary Strategies (ES) instead of gradient-based RL, the low-level MPC can be treated as a black box, which proved more robust for this architecture.
Experimental Results
The framework was evaluated using the PyBullet physics engine for simulation and the Unitree A1 quadruped for hardware validation.
Emergent Gait Transitions
A key finding is that a simple energy minimization reward is sufficient for complex gaits to emerge. As the robot increases its speed, it automatically transitions from walking (< 0.9 m/s) to trotting (0.9–1.8 m/s) and finally to "fly-trotting" (> 1.8 m/s). Interestingly, the robot did not discover galloping; analysis revealed that for this specific morphology, galloping was 30% less efficient than trotting.
Energy Efficiency and Robustness
The learned policy consistently outperformed hand-tuned baseline controllers in terms of energy efficiency.
The robot reached speeds of 2.5 m/s (approximately 5 body lengths per second) and demonstrated "zero-shot" transfer to real-world environments, successfully navigating grass, carpet, and 8cm-high steps without additional fine-tuning.
Discussion and Future Work
The hierarchical approach acts as a bridge between high-level behavioral learning and low-level physical constraints. The MPC provides a safety net that allows the RL policy to be simpler and more transferable. Future work will explore expanding the high-level policy to control foot placement and base pose, potentially moving toward a bi-level optimization for tighter integration. The results suggest that future robotic designs should be co-optimized with their controllers to achieve animal-like agility and efficiency.
Summary updated