Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning

Jiyuan Shi, Xinzhe Liu, Dewei Wang, Ouyang Lu, Sören Schwertfeger, Chi Zhang, Fuchun Sun, Chenjia Bai, Xuelong Li

TL;DR

The Adversarial Locomotion and Motion Imitation (ALMI) framework achieves robust whole-body humanoid control by decoupling upper-body motion tracking from lower-body locomotion and training them as adversaries in a zero-sum game. This approach enables full-sized humanoid robots to execute complex, expressive upper-body movements while maintaining rock-solid walking stability in the real world. Along with the framework, the authors release ALMI-X, a massive dataset of over 80,000 language-annotated whole-body trajectories to jumpstart humanoid foundation models.

What Problem Does This Paper Address?

Controlling a full-sized humanoid robot is a notorious challenge due to its high Degrees of Freedom (DoF) and inherently unstable floating-base dynamics. Traditional reinforcement learning (RL) methods typically train a single, unified policy to handle whole-body control. However, these methods often treat the entire body uniformly, ignoring the fundamentally distinct roles of the upper body (precise motion tracking and manipulation) and the lower body (robust locomotion and balance maintenance).

When a robot attempts to mimic expressive human gestures, the rapid shifts in upper-body mass and inertia create severe physical disturbances that destabilize the legs. As a result, unified policies often prioritize motion tracking at the expense of balance, leading to frequent falls during real-world deployment.

Methods

ALMI tackles this challenge by formulating whole-body control as a two-player zero-sum Markov game. Instead of a single policy, ALMI splits control into two specialized policies that undergo iterative adversarial training:

  1. Lower-Body Locomotion Policy ($\pi^l$): Aims to follow velocity commands while treating upper-body movements as adversarial disturbances to be resisted.
  2. Upper-Body Motion-Imitation Policy ($\pi^u$): Aims to precisely track reference human motions despite adversarial balance disruptions caused by the moving lower body.
Overview of the ALMI adversarial training framework.

To make this minimax optimization computationally practical, the authors introduce two key innovations:

  • Simplified Formulation: Instead of optimizing both policies simultaneously in parameter space (which is highly unstable), they freeze one policy and sample adversarial commands from the environment's command space.
  • Dual Curriculum Mechanism: The framework automatically scores the difficulty of upper-body motions based on the robot's "survival length" (how long it stands before falling). It introduces mild disturbances early in training and progressively scales up the amplitude ($\alpha_s$) and difficulty window ($\alpha_d$) of the adversarial motions as the lower body's robustness improves.

Datasets

The paper utilizes and introduces several key datasets:

  • AMASS & CMU MoCap: Used as the source of reference human motion clips for training and evaluation.
  • ALMI-X (New): A newly introduced large-scale whole-body control dataset featuring over 80,000 trajectories collected in MuJoCo simulations using the trained ALMI policy. It pairs omnidirectional lower-body velocity commands with 30 distinct categories of upper-body motions (e.g., waving, punching, vacuuming). Crucially, each trajectory is annotated with structured language descriptions (e.g., "moving backward slowly and wave the left hand"), making it a valuable benchmark for training language-guided robotic foundation models.
Figure 2: ALMI-X dataset and foundation model training. (a) ALMI-X features motion target and velocity command for the learned policies, combining with language description. (b) The foundation model learns P(ˆai+1|s≤i, a≤i, T ) from a segment of state-action pairs via causal attention. In inference, the last action is executed based on the history and obtains the true state for next steps.

Key Insights and Ideas

  • Decoupling via Adversarial Play: Rather than manually tuning complex reward functions to balance walking vs. gesturing, framing the two body halves as adversaries forces the policies to naturally co-adapt. The legs learn to counter inertial shifts, while the arms learn to stabilize tracking despite a moving base.
  • Command-Space Disturbance Sampling: Swapping parameter-space adversarial optimization for structured command-space sampling makes the zero-sum game highly stable and computationally efficient, converging in just three iterative rounds (~17 hours).
  • Survival Length as a Curriculum Metric: Using the robot's time-to-fall as a direct metric for motion difficulty eliminates the need for labor-intensive, manual reward shaping for complex terrains and disturbances.

Key Results

The authors evaluated ALMI against state-of-the-art baselines (including ExBody, ExBody2, and OmniH2O) in both simulation and real-world hardware using the full-sized Unitree H1-2 robot.

  • Superior Robustness: On the "Hard" evaluation setting (which includes uneven terrain and external pushing forces), ALMI achieved a 97.23% survival rate, significantly outperforming ExBody's 87.78%.
  • High Tracking Precision: ALMI consistently minimized both linear velocity errors ($E_{vel}$) for locomotion and joint position errors ($E_{jpe}$) for upper-body tracking across all difficulty tiers.
  • Zero-Shot Sim-to-Real Transfer: Thanks to robust domain randomization, the policies transferred directly to the real Unitree H1-2 robot without any additional tuning. The robot successfully executed dynamic tasks like swinging a tennis racket, carrying a 5kg box, and vacuuming the floor while walking stably.

Limitations and Future Work

Despite its strengths, the authors note a few limitations:

  • Highly Dynamic Coordination: Because the framework decouples the upper and lower body, it performs suboptimally on tasks requiring tight, highly dynamic whole-body synchronization, such as dancing.
  • Foundation Model Scaling: While a Transformer decoder trained on a subset of ALMI-X successfully achieved language-guided control (e.g., waving while walking), performance degraded when scaling to the full, highly diverse dataset. Improving data efficiency and model architectures for complete whole-body foundation models remains an open area for future exploration.

Summary updated

Abstract

Humans exhibit diverse and expressive whole-body movements. However, attaining human-like whole-body coordination in humanoid robots remains challenging, as conventional approaches that mimic whole-body motions often neglect the distinct roles of upper and lower body. This oversight leads to computationally intensive policy learning and frequently causes robot instability and falls during real-world execution. To address these issues, we propose Adversarial Locomotion and Motion Imitation (ALMI), a novel framework that enables adversarial policy learning between upper and lower body. Specifically, the lower body aims to provide robust locomotion capabilities to follow velocity commands while the upper body tracks various motions. Conversely, the upper-body policy ensures effective motion tracking when the robot executes velocity-based movements. Through iterative updates, these policies achieve coordinated whole-body control, which can be extended to loco-manipulation tasks with teleoperation systems. Extensive experiments demonstrate that our method achieves robust locomotion and precise motion tracking in both simulation and on the full-size Unitree H1 robot. Additionally, we release a large-scale whole-body motion control dataset featuring high-quality episodic trajectories from MuJoCo simulations deployable on real robots. The project page is https://almi-humanoid.github.io.
BibTeX
@inproceedings{shi2025adversarial,
  author = {Jiyuan Shi and Xinzhe Liu and Dewei Wang and Ouyang Lu and Sören Schwertfeger and Chi Zhang and Fuchun Sun and Chenjia Bai and Xuelong Li},
  title = {{Adversarial Locomotion and Motion Imitation for Humanoid Policy Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=hVoIz6xD9Q},
}
Figure