Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer

Xinyang Gu, Yen-Jen Wang, Jianyu ChenarXiv 2024 · arXiv · OpenReview

Abstract

Humanoid-Gym is an easy-to-use reinforcement learning (RL) framework based on Nvidia Isaac Gym, designed to train locomotion skills for humanoid robots, emphasizing zero-shot transfer from simulation to the real-world environment. Humanoid-Gym also integrates a sim-to-sim framework from Isaac Gym to Mujoco that allows users to verify the trained policies in different physical simulations to ensure the robustness and generalization of the policies. This framework is verified by RobotEra's XBot-S (1.2-meter tall humanoid robot) and XBot-L (1.65-meter tall humanoid robot) in a real-world environment with zero-shot sim-to-real transfer. The project website and source code can be found at: https://sites.google.com/view/humanoid-gym/.

Introduction

Training humanoid robots is notoriously difficult due to their inherent instability, high degrees of freedom (DoF), and extreme sensitivity to the "sim-to-real gap." While massively parallel RL in simulation has accelerated robot learning, policies that perform perfectly in a simulator often fail on physical hardware due to unmodeled dynamics, sensor noise, and latency. Humanoid-Gym addresses these challenges by providing an accessible, end-to-end open-source framework specifically tailored for robust humanoid locomotion.

Methodology: The Sim-to-Sim-to-Real Pipeline

The Humanoid-Gym framework introduces a multi-stage pipeline designed for maximum robustness:

  1. Massively Parallel Training: Using Nvidia Isaac Gym, the framework trains policies using Proximal Policy Optimization (PPO). To enhance learning, the authors utilize an Asymmetric Actor-Critic approach, where the Critic has access to privileged information (e.g., exact friction or ground truth velocities) that the Actor will not have in the real world.
  2. Sim-to-Sim Validation: Before hardware deployment, policies are tested in MuJoCo. While Isaac Gym is optimized for training speed, MuJoCo provides higher-fidelity physics. The authors meticulously calibrated MuJoCo to match real-world dynamics, providing a critical safety check.
  3. System Design & Rewards: The control policy operates at 100Hz, while the internal PD controller runs at 1000Hz. The framework uses specialized reward functions for velocity tracking, gait symmetry, and energy efficiency, alongside heavy domain randomization to ensure the policy is resilient to simulator discrepancies.
The Humanoid-Gym pipeline: training in Isaac Gym, validation in MuJoCo, and deployment in the Real World.

Experimental Results

The framework was verified using two distinct humanoid robots from RobotEra: the 1.2-meter tall XBot-S (32 DoF) and the 1.65-meter tall XBot-L (60 DoF).

Zero-Shot Success and Robustness

Policies trained entirely in simulation were deployed directly onto the XBot-S and XBot-L robots without manual fine-tuning. The robots successfully traversed both flat and uneven real-world terrains, maintaining stability despite the complexity of their frames. A key insight was that post-calibration, the leg swing trajectories in MuJoCo were nearly identical to real-world measurements, validating the sim-to-sim approach as a reliable proxy for real-world performance.

Real-world deployment of the XBot-L humanoid robot demonstrating stable locomotion.

Conclusion and Future Work

Humanoid-Gym demonstrates that meticulous system design and domain randomization can eliminate the need for time-consuming sim-to-real fine-tuning. While currently optimized for lower-body locomotion, future work will explore complex upper-body manipulation and automated system identification to further streamline the calibration process. The authors have released the full codebase to the community to lower the barrier to entry for humanoid research.

Summary updated

Abstract

Humanoid-Gym is an easy-to-use reinforcement learning (RL) framework based on Nvidia Isaac Gym, designed to train locomotion skills for humanoid robots, emphasizing zero-shot transfer from simulation to the real-world environment. Humanoid-Gym also integrates a sim-to-sim framework from Isaac Gym to Mujoco that allows users to verify the trained policies in different physical simulations to ensure the robustness and generalization of the policies. This framework is verified by RobotEra's XBot-S (1.2-meter tall humanoid robot) and XBot-L (1.65-meter tall humanoid robot) in a real-world environment with zero-shot sim-to-real transfer. The project website and source code can be found at: https://sites.google.com/view/humanoid-gym/.
BibTeX
@misc{gu2024humanoid,
  author = {Xinyang Gu and Yen-Jen Wang and Jianyu Chen},
  title = {{Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer}},
  year = {2024},
  eprint = {2404.05695},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.05695},
}
Figure