TL;DR
HUMANUP is a novel two-stage reinforcement learning framework that enables human-sized humanoid robots to autonomously recover from falls across diverse, challenging real-world terrains. By decoupling trajectory discovery from hardware deployment constraints, the system successfully transfers learned getting-up and rolling-over policies to the Unitree G1 robot, significantly outperforming hand-crafted commercial controllers.
What Problem Does This Paper Address?
While reinforcement learning (RL) has revolutionized humanoid locomotion (like walking and running), automatic fall recovery remains a major bottleneck for deploying these robots in the real world. If a humanoid falls, it can land in an unpredictable configuration on highly irregular terrain.
Standard RL locomotion tricks fail here for three reasons:
- Non-periodic behavior: Locomotion relies on cyclic, predictable left-right stepping patterns. Getting up requires figuring out an entirely unstructured sequence of contacts.
- Richness in contact: Unlike walking, which only involves foot-ground contact, getting up requires the robot to use its entire body (arms, torso, knees) to interact with the environment.
- Reward sparsity: In locomotion, velocity tracking provides immediate, dense feedback. When getting up, a robot often has to make "negative progress" first—such as tilting its torso down for several seconds—before it can successfully hoist itself up, making standard reward structures ineffective.
Methods
The authors introduce HUMANUP, a two-stage training framework that establishes a clever curriculum to bridge the sim-to-real gap:
- Stage I: Discovery Policy: This stage focuses purely on solving the task in an idealized environment. The robot learns a getting-up trajectory using a simplified collision mesh, a canonical starting pose, and very weak control regularization. This allows the RL agent to freely explore and discover successful motions without being penalized for jerky movements or high torques.
- Stage II: Deployable Policy: The discovered trajectory is slowed down (by 8×) and used as a reference target. In this stage, the robot learns to track this motion under strict real-world constraints: a full whole-body collision mesh, heavy control regularization (penalizing joint jitter and excessive torque), posture randomization, and domain/terrain randomization.
Datasets
To train and evaluate the system's robustness, the authors curated a simulated dataset $\mathcal{P}$ consisting of:
- 20,000 supine poses (lying face up)
- 20,000 prone poses (lying face down)
These configurations were generated in Isaac Gym by randomizing the robot's initial joint angles, dropping it from a height of 0.5 meters, and letting it settle for 10 seconds to resolve self-collisions. 10,000 poses from each set were held out exclusively for evaluation.
Key Insights and Ideas
- The "Hard-to-Easy" and "Easy-to-Hard" Curriculum: Splitting the pipeline into two stages creates a dual curriculum. Moving from Stage I to Stage II represents a hard-to-easy transition in terms of task difficulty (shifting from open-ended task discovery to simple motion tracking), but an easy-to-hard transition for regularization and environmental variability. Trying to learn both simultaneously fails completely.
- Soft Symmetry Rewards: Instead of forcing strict mathematical bilateral symmetry (which limits the robot's degrees of freedom and generalization), HUMANUP uses a soft symmetry reward. This guides the agent toward balanced motions while allowing it to break symmetry when dealing with uneven terrains.
- Leveraging Full URDF Geometry: Training Stage II with the official, high-fidelity whole-body URDF mesh—rather than a simplified capsule approximation—proved essential for real-world transfer, as it accurately captured the complex contact dynamics of the robot's shell against the ground.
Key Results
The learned policies were deployed directly on a real-world Unitree G1 humanoid robot across six diverse terrains: concrete, brick, stone tiles, muddy grass, grass slopes (~10°), and a snowfield.
- High Success Rates: In simulation, HUMANUP achieved a 95.34% success rate from supine poses and 92.10% from prone poses. In the real world, it achieved an overall 78.3% getting-up success rate from supine positions and 98.3% for rolling over.
- Outperforming the Manufacturer Baseline: The default, hand-crafted controller provided by Unitree only succeeded on flat, firm concrete. HUMANUP successfully navigated bumps, slopes, and slippery snow.
- Efficiency and Safety: HUMANUP stands up in about 6 seconds—over 2× faster than the manufacturer's controller (11 seconds). Furthermore, by utilizing the robot's stronger leg motors rather than forcing it to push up heavily with its arms, HUMANUP kept motor temperatures significantly lower, preventing hardware damage.
Limitations and Future Work
Despite its success, the authors note several limitations:
- Unnatural Behaviors: Because the reward function is under-specified, the robot occasionally exhibits "reward hacking" behaviors, such as raising an arm awkwardly in the air for balance.
- Simulation Dependency: The framework relies heavily on high-frequency (1 kHz) physics simulation in Isaac Gym; modeling even more complex interactions (like deformable mud or deep snow) remains a challenge.
- Complex Terrains: The current policy has not yet been extended to structured, highly uneven environments like stairs or steep, jagged obstacles.
Summary updated