Learning Quadruped Locomotion Using Differentiable Simulation

Yunlong Song, Sangbae Kim, Davide Scaramuzza

Introduction

Traditional model-free reinforcement learning (RL) for legged robots is notoriously data-hungry. Algorithms like Proximal Policy Optimization (PPO) rely on "zeroth-order" gradients—a trial-and-error approach requiring thousands of parallel simulations to find a stable policy. This paper introduces a novel framework for training quadruped robots using differentiable simulation (DiffSim), enabling complex locomotion skills to be learned in minutes rather than hours.

While DiffSim provides "first-order" analytical gradients—akin to feeling the slope of a hill to find the peak—it often struggles with the discontinuous dynamics of walking, such as sudden foot-ground impacts. These discontinuities create a messy optimization landscape. The proposed method bypasses this by combining a high-fidelity, non-differentiable simulator for forward physics with a simplified, differentiable surrogate model for gradient backpropagation.

Methodology: Hybrid Differentiable Simulation

The core innovation is a hybrid approach that leverages the accuracy of standard physics engines while maintaining the gradient flow of differentiable models. The system architecture decouples the complex whole-body dynamics from the gradient-friendly surrogate model.

  1. Surrogate Dynamics: Instead of differentiating through complex whole-body contacts, the system uses a smooth Single Rigid-Body Dynamics (SRBD) model for the robot's body.
  2. High-Fidelity Forward Pass: The actual robot state is updated using IsaacGym, a non-differentiable but highly accurate physics engine.
  3. State Alignment: To prevent the simplified model from drifting, the state of the differentiable model is "aligned" with the high-fidelity simulator at every timestep. This ensures gradients are grounded in realistic physics.
  4. Short-Horizon Training: Simulation is truncated into short segments (e.g., 24 steps) for backpropagation to prevent gradient explosion or vanishing.
System architecture showing the hybrid differentiable simulation framework and state alignment process.

Experimental Evaluation

The framework was evaluated on a double integrator toy example and locomotion tasks for the Mini Cheetah robot. Training was conducted on flat ground and challenging terrains including slopes and obstacles.

Key Results

The DiffSim approach demonstrated unprecedented sample efficiency and robust performance compared to standard RL baselines:

  • Efficiency: A single simulated robot learned to walk on a laptop in minutes. With GPU parallelization, it mastered diverse gaits (trot, pace, bound, gallop) in similar timeframes.
  • Sample Efficiency: DiffSim significantly outperformed PPO. In low-sample regimes (e.g., 4 or 16 robots), PPO failed to learn stable gaits, while DiffSim reached high reward levels rapidly.
  • Zero-Shot Transfer: Policies were transferred directly to a physical Mini Cheetah without fine-tuning. The robot successfully navigated varied terrains and resisted external disturbances, proving the simulation's gradients were grounded in physical reality.
Comparison of training progress, trajectories, and control inputs between DiffSim, PPO, and LQR.

Discussion and Future Work

The "aha moment" of this research is that a perfectly differentiable world is unnecessary for first-order gradient benefits. A "good enough" differentiable approximation, constantly corrected by a high-fidelity simulator, provides a cleaner learning signal than pure RL.

However, limitations remain. The current system relies on the Raibert heuristic for foot placement and predefined gait schedules. Future work aims to integrate vision-based control and remove reliance on predefined schedules to allow for more emergent, adaptive behaviors.

Summary updated

Abstract

This work explores the potential of using differentiable simulation for learning robot control. Differentiable simulation promises fast convergence and stable training by computing low-variance first-order gradients using the robot model. Still, so far, its usage for legged robots is limited to simulation. The main challenge lies in the complex optimization landscape of robotic tasks due to discontinuous dynamics. This work proposes a new differentiable simulation framework to overcome these challenges. The key idea involves decoupling the complex whole-body simulation, which may exhibit discontinuities due to contact into two separate continuous domains. Subsequently, we align the robot state resulting from the simplified model with a more precise, non-differentiable simulator to maintain sufficient simulation accuracy. Our framework enables learning quadruped walking in simulation in minutes without parallelization. When augmented with GPU parallelization, our approach allows the quadruped robot to master diverse locomotion skills on challenging terrains in minutes. We demonstrate that differentiable simulation outperforms a reinforcement learning algorithm (PPO) by achieving significantly better sample efficiency while maintaining its effectiveness in handling large-scale environments. Our policy achieves robust locomotion performance in the real world zero-shot.
BibTeX
@inproceedings{song2024learning,
  author = {Yunlong Song and Sangbae Kim and Davide Scaramuzza},
  title = {{Learning Quadruped Locomotion Using Differentiable Simulation}},
  booktitle = {Conference on Robot Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=XopATjibyz},
}
Figure