TL;DR
FORGE introduces a two-stage reinforcement learning framework that enables humanoid robots to perform a wide array of agile maneuvers—from high-speed running to one-legged hopping—using a single unified policy. By coupling a kinematically-aware reduced-order model (ROM) for online reference generation with a full-order tracking controller, the system achieves impressive zero-shot sim-to-real transfer on the Unitree G1 humanoid.
What Problem Does This Paper Address?
Humanoid locomotion is notoriously difficult because these robots are high-dimensional, underactuated, and governed by nonlinear hybrid dynamics. Traditionally, researchers choose between two paths:
- Model-based Optimization: Using full-order models (FOM) to plan every joint movement is computationally expensive and often too slow for real-time adjustments.
- Reinforcement Learning (RL): While robust, training a single RL policy to handle a broad range of commands (like sprinting, turning, and hopping) without "mode-switching" logic is a significant challenge.
Furthermore, many RL methods rely on large datasets of human motion capture. However, human data doesn't always translate well to robot proportions, leading to "kinematically infeasible" references—motions the robot's joints simply cannot physically execute. FORGE aims to bridge this gap by generating high-quality, feasible references online without needing human data.
Methods
The authors propose a two-stage training architecture:
Stage 1: Reference Generation Instead of a simple point-mass model, the authors train a policy in a custom "ROM simulator." This simulator combines Single Rigid Body (SRB) dynamics (treating the robot as a lumped mass with fixed inertia) with the full humanoid kinematics. By including the robot's actual joint limits and geometry in this simplified stage, the policy learns to generate reference motions that are "kino-dynamically consistent"—meaning they respect both the laws of physics and the robot's physical structure.
Stage 2: Whole-Body Tracking The frozen policy from Stage 1 is used to generate references online. A second policy is then trained on the full-order dynamics (the "real" physics) to track these references. This stage focuses on closing the gap between the simplified ROM physics and the complex reality of limb inertia and contact forces.
Datasets
The framework does not rely on external datasets like human motion capture. Instead, it uses:
- Custom ROM Simulator: A numerical environment for Stage 1 training that evaluates contact and collision detection.
- MuJoCo: Used for visualization and Stage 2 training on the full-order model.
- Unitree G1 Model: The specific kinematic and dynamic parameters of the Unitree G1 humanoid were used to ensure the generated motions were tailored to the hardware.
Key Insights and Ideas
- Feasibility by Design: By incorporating full kinematics into the ROM training, the reference generator "knows" what the legs can and cannot do. This prevents the tracker from trying to follow impossible commands.
- Unified Command Space: The framework doesn't use separate policies for walking vs. hopping. It treats hopping as just another command ($h_F$) that can be layered onto a velocity command.
- Gait Phasing: The use of a monotonic phasing variable ($\phi$) allows the robot to seamlessly speed up or slow down its gait, providing a smooth transition between walking and running.
Key Results
The researchers demonstrated the system on a Unitree G1 humanoid with zero-shot transfer (no tuning on the physical robot):
- Speed: Achieved longitudinal velocities from -2.0 m/s (backward) to 2.8 m/s (forward). In simulation, it reached up to 4.0 m/s.
- Agility: Handled lateral velocities of $\pm 1.0$ m/s and yaw rates of $\pm 3.0$ rad/s.
- Versatility: The robot successfully transitioned from walking to one-legged hopping on challenging terrain like sand.
- Robustness: Even though trained only on flat ground, the robot could run on uneven, sloped grassy terrain.
Limitations and Future Work
While the results are impressive, the authors noted a few constraints:
- Hardware Limits: Some high-agility maneuvers (like 4.0 m/s sprinting or sustained hopping) were only demonstrated in simulation because they approached the maximum torque and power capabilities of the G1 hardware.
- Terrain Awareness: The current policy was trained on flat ground. While it generalized well to grass and sand, explicit terrain-aware training could further improve robustness.
- Future Directions: The team plans to conduct ablation studies to see which components (like the specific reward terms) contribute most to the performance and to compare FORGE against standard model-based controllers.
Summary updated