TL;DR
ReLoco is a reinforcement learning framework that unifies humanoid locomotion and fall recovery into a single, seamless proprioceptive policy. By using behavior-conditioned adversarial motion priors and a simulation-based motion refinement stage, it enables full-sized humanoid robots to walk, run, and autonomously recover from arbitrary falls without the need for manual controller switching.
What Problem Does This Paper Address?
Humanoid robots are inherently unstable. While significant progress has been made in steady-state locomotion (walking and running), a single fall often spells the end of a mission. Traditionally, "getting up" and "walking" are treated as separate problems solved by separate controllers.
The challenge lies in the vast difference between these behaviors: walking requires precise velocity tracking and upright stability, while recovery involves complex ground contacts, rolling, and large torso rotations. Bridging the gap between a fallen state and a commanded gait requires a policy that understands both regimes and, crucially, the transition between them. ReLoco aims to create a "generalist" controller that can handle the entire pipeline—from a face-plant back to a full sprint—using only the robot's internal sensors (proprioception).
Methods
The ReLoco framework operates in two main stages:
1. Simulation-Based Motion Refinement
Raw human motion capture (mocap) data often contains physical impossibilities when retargeted to a robot, such as self-collisions or "floating" feet. ReLoco first uses an offline tracking policy to "clean" these motions. This policy attempts to mimic the human mocap while adhering to the robot's actual physics and actuator limits. The result is a library of robot-executed references that provide a much cleaner signal for the next stage of training.
2. Behavior-Conditioned Adversarial Motion Priors (AMP)
The core of ReLoco is a unified policy trained via Reinforcement Learning (PPO). To ensure the robot moves naturally, the authors use Adversarial Motion Priors (AMP), where a discriminator network learns to distinguish between "expert" motions and the robot's current behavior.
The "secret sauce" here is behavior conditioning:
- Posture-Based Routing: During training, a simple rule based on torso height and alignment determines if the robot is "upright" or "fallen."
- Multi-Head Discriminator: Instead of one output, the discriminator has two heads—one for locomotion style and one for recovery style. This prevents the styles from "bleeding" into each other, ensuring the robot doesn't try to walk like it's getting up.
- Force Curriculum: To jumpstart the difficult task of learning to stand up, the authors apply a temporary "hand of god" upward force to the robot's head during early training, which gradually decays as the robot becomes self-sufficient.
Datasets
- Human Mocap Assets: A collection of walking, jogging, running, and get-up motions.
- BONES-SEED: A publicly available skeletal embodiment dataset used for diverse human motion sequences.
- Self-Collected Data: The authors supplemented existing datasets with their own human motion captures to ensure a wide variety of recovery maneuvers.
Key Insights and Ideas
- Clean Data Matters: Using "refined" robot-executed motions instead of raw retargeted mocap significantly boosts performance. As shown in Figure 2, the refined references lead to higher rewards and a much faster climb in get-up success rates.
- Conditioning is Crucial: Without behavior conditioning, a single-output discriminator struggles to balance the competing demands of upright walking and ground recovery. Separating the "style" heads allows the policy to achieve 14.7% lower velocity tracking error while maintaining high recovery success.
- The "Hand of God" Works: The force curriculum (Figure 3) is essential for exploration. Without it, the robot often gets stuck in "local minima" (like wiggling on the floor) and never discovers the complex sequence of movements needed to stand up.
Key Results
The authors deployed ReLoco on the Tiangong 3.0, a 64.5-kg full-sized humanoid.
- High Success Rates: In simulation, the policy achieved a 96.80% success rate for independent get-up tasks and 96.96% for push-recovery (recovering from a fall and resuming the commanded velocity).
- Real-World Robustness: The robot successfully navigated diverse terrains, including stone tiles, grass, and slopes (Figure 4). It reached treadmill speeds of up to 9.3 km/h.
- Autonomous Recovery: In hardware tests, the robot recovered from 34 out of 34 falls caused by external kicks, handling prone, supine, and side-lying positions with an average get-up time of 2.64 seconds (Figure 5).
Limitations and Future Work
- Blind Recovery: The current policy is purely proprioceptive. While impressive, it cannot "see" obstacles or terrain during recovery, which might lead to collisions in cluttered environments.
- Reference Dependency: The quality of the final behavior is heavily dependent on the quality and diversity of the initial mocap library.
- Future Directions: The authors suggest integrating vision for navigation and more complex terrain handling, as well as expanding the motion library to include even more agile "protection" behaviors (like tucking and rolling) to minimize impact damage during falls.
Summary updated