Think on your feet: Seamless Transition between Human-like Locomotion in Response to Changing Commands

Huaxing Huang, Wenhao Cui, Tonghe Zhang, Shengtao Li, Jinchao Han, Bangyu Qin, Tianchu Zhang, Liang Zheng, Ziyang Tang, Chenxu Hu, Ning Yan, Jiahao Chen, Shipu Zhang, Zheyuan Jiang

TL;DR

Humanoid robots often face a strict trade-off between moving naturally (via imitation learning) and responding agilely to dynamic commands (via reinforcement learning). This paper introduces a framework that bridges this gap by combining a Hybrid Internal Model, Wasserstein divergence, and curiosity-driven exploration. The result is a control policy that enables humanoids to transition seamlessly between diverse, human-like gaits while strictly adhering to changing velocity commands in the real world.

What Problem Does This Paper Address?

Controlling humanoid robots is notoriously difficult due to their high degrees of freedom, large mass, and high center of gravity. Standard Reinforcement Learning (RL) can produce stable locomotion, but without tight constraints, it often results in unnatural, quirky gaits that cause high mechanical wear and energy expenditure.

To fix this, researchers use Imitation Learning (IL)—such as Adversarial Motion Priors (AMP)—to force robots to mimic human Motion Capture (MoCap) data. However, pure imitation learning suffers from a major flaw: it struggles to generalize. If a robot is only trained to mimic a specific walking or running clip, it cannot easily adapt to continuously changing velocity commands or figure out how to execute intermediate movements (like transitioning smoothly from a slow walk to a sudden sprint) that aren't explicitly in the dataset. The authors of this paper set out to solve this exact problem: achieving fluid, human-like locomotion that remains highly responsive to dynamic, real-time user commands.

Methods

The authors build upon the classical AMP framework by introducing three key architectural enhancements:

  1. Hybrid Internal Model (HIM): To improve mobile stability, an estimator network extracts structured velocity estimates and latent representations of hidden states from historical observations. This gives the policy a clearer understanding of the relationship between the user's input commands and the robot's physical state.
  2. Wasserstein GAN Divergence (WGAN-div): Traditional GAN-based discriminators frequently suffer from "mode collapse," where the generator settles on a single, stiff gait pattern. By replacing the standard loss with a Wasserstein divergence criterion, the framework ensures smoother gradients and allows the robot to capture a multi-peak distribution of diverse joint movements.
  3. Curiosity Hash-Net: To prevent the RL policy from getting trapped in local minima during training, the authors introduce a count-based curiosity reward ($r^c_t$). A neural network maps continuous states to binary hashing strings, rewarding the robot for exploring less-visited state-action spaces.

The entire pipeline is optimized using Proximal Policy Optimization (PPO) and robustified via extensive domain randomization to bridge the sim-to-real gap.

Fig. 3: Illustration of the proposed human-like locomotion learning framework. The estimator extracts information from past observations, producing a velocity estimate with internal state representation. The velocity estimate ensures mobile stability, while contrastive learning promotes future observation prediction. Imitation learning secures human-like gaits, and a curiosity bonus fosters exploration. The bottom-left block is employed in real-world deployment.

Datasets

The framework utilizes human Motion Capture (MoCap) reference data containing basic straight walking and running sequences. The system was trained and evaluated across two major physics simulators:

  • Isaac Gym (for massive parallel training)
  • MuJoCo (for zero-shot validation)

The final policies were deployed directly onto physical hardware, specifically the Noetix N1 (an 18-DoF humanoid weighing 23 kg) and the Dora robot.

Key Insights and Ideas

  • Deconstructing Expert Motions: One of the most compelling findings is that the combination of WGAN-div and RL allows the system to break down expert human motion sequences into fine-grained "basic action units." The robot can then reassemble these units to perform completely unseen motions—such as walking backward, taking sidesteps, or spinning while running—even though those specific actions were never in the MoCap training data.
  • Mitigating the HIM Penalty: While incorporating a Hybrid Internal Model (HIM) drastically improves velocity tracking, it initially degrades the "human-likeness" of the gait because the RL system prioritizes task completion over style. However, the authors discovered that adding WGAN-div and curiosity rewards effectively mitigates this penalty, correcting joint positions to preserve the natural human flavor.
  • Curiosity Shifts the Distribution: Rather than just adding random noise to exploration, the curiosity bonus actively shifts the style reward distribution toward higher-reward, more stable regions during the training process.

Key Results

  • Superior Velocity Tracking: When tested against abruptly changing velocity commands, pure IL methods failed to keep up. The authors' full-stack approach (AMPw-HIM+) achieved a remarkably low tracking error of 8.5 cm/s, compared to the baseline AMP's error of 22.9 cm/s.
  • Seamless Zero-Shot Transfer: The exact same policy and Proportional-Derivative (PD) controller parameters trained in simulation were deployed directly onto the real-world Noetix N1 robot. The robot successfully walked, ran, and stopped on command across both indoor floors and challenging outdoor terrains (such as slopes and bumpy surfaces).
Fig. 1: Comprehensive demonstration of Noetix robot N1’s locomotion skills learnt from the proposed method. The robot exhibits seamless and continuous transfer between highly human-like motion sets, accelerating from walking to running then coming to a full stop. Top to down: performance in simulator Isaac gym, Mujoco, and the real world. Parameters of the PD controller and the driving frequency are consistent in the three groups.

Limitations and Future Work

While the framework demonstrates impressive robustness on bumpy ground and descending slopes, it has not yet been adapted to highly structured, discrete obstacles like stairs. Additionally, the current model focuses entirely on lower-body locomotion. The authors note that a promising future direction is to scale this framework to whole-body control, coordinating these fluid gaits with upper-body hand manipulation tasks or more dynamic aerial phases like hopping.

Summary updated

Abstract

While it is relatively easier to train humanoid robots to mimic specific locomotion skills, it is more challenging to learn from various motions and adhere to continuously changing commands. These robots must accurately track motion instructions, seamlessly transition between a variety of movements, and master intermediate motions not present in their reference data. In this work, we propose a novel approach that integrates human-like motion transfer with precise velocity tracking by a series of improvements to classical imitation learning. To enhance generalization, we employ the Wasserstein divergence criterion (WGAN-div). Furthermore, a Hybrid Internal Model provides structured estimates of hidden states and velocity to enhance mobile stability and environment adaptability, while a curiosity bonus fosters exploration. Our comprehensive method promises highly human-like locomotion that adapts to varying velocity requirements, direct generalization to unseen motions and multitasking, as well as zero-shot transfer to the simulator and the real world across different terrains. These advancements are validated through simulations across various robot models and extensive real-world experiments.
BibTeX
@inproceedings{huang2025think,
  author = {Huaxing Huang and Wenhao Cui and Tonghe Zhang and Shengtao Li and Jinchao Han and Bangyu Qin and Tianchu Zhang and Liang Zheng and Ziyang Tang and Chenxu Hu and Ning Yan and Jiahao Chen and Shipu Zhang and Zheyuan Jiang},
  title = {{Think on your feet: Seamless Transition between Human-like Locomotion in Response to Changing Commands}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2025},
}
Figure