Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Ye Yuan, Kris Kitani

TL;DR

Residual Force Control (RFC) bridges the "dynamics mismatch" gap by adding learnable external forces to humanoid control, enabling the imitation of highly agile behaviors like ballet. When paired with a dual-policy framework, it allows for the first stable, long-term motion synthesis from large-scale datasets like Human3.6M without requiring manual task guidance or user input.

What Problem Does This Paper Address?

Creating virtual humans that move realistically is a cornerstone of animation, gaming, and VR. While physics-based methods using Reinforcement Learning (RL) produce motions free of artifacts like foot-sliding, they often fail to capture highly agile skills (e.g., ballet) or long-term sequences with complex transitions.

The authors identify the core culprit as dynamics mismatch: the simplified humanoid models used in simulations are not exact replicas of complex real humans. Consequently, many real-world motions are physically impossible for the model to execute under standard physics. Furthermore, synthesizing long-term motion typically requires tedious manual task definitions or constant user interaction to keep the agent from falling over.

Methods

The paper introduces two primary innovations:

  1. Residual Force Control (RFC): The authors augment the humanoid's action space with external "residual forces." These forces act as a learnable, time-varying correction to the humanoid's dynamics, effectively allowing the agent to "cheat" slightly to overcome model limitations. They propose two flavors:
  2. RFC-Explicit: Models specific force vectors and contact points (e.g., at the feet).
  3. RFC-Implicit: Models the total joint torques at the root, which is more computationally efficient and requires fewer assumptions. To keep the motion natural, these forces are heavily regularized so the agent only uses them when absolutely necessary.

  4. Dual-Policy Control Framework: To handle long-term synthesis, the system uses two policies in tandem:

  5. Kinematic Policy: A Conditional Variational Autoencoder (CVAE) that predicts future multi-modal motions based on past states, serving as the "intent" of the character.
  6. RFC-based Control Policy: This policy takes the kinematic prediction as a reference and uses RFC to physically execute the motion in the simulator.

Datasets

The authors evaluated their approach on three major benchmarks:

  • CMU Motion Capture (MoCap): Used for short, highly agile clips including ballet (pirouettes, arabesques) and acrobatics (backflips, jump kicks).
  • Human3.6M: A large-scale dataset used to test the dual-policy framework's ability to synthesize diverse, long-term motions.
  • EgoMocap: A smaller dataset used for further validation of motion synthesis.

Key Insights and Ideas

  • The "Invisible Hand": Residual forces can be thought of as invisible hands that assist the humanoid. By allowing these small physical "corrections," the agent can successfully learn complex balance-heavy moves that were previously impossible.
  • Intent vs. Execution: By separating the prediction of motion (kinematics) from the execution of motion (physics), the system avoids the instability common in long-term RL tasks. The kinematic policy provides a "good guess" for the target joint angles, which significantly eases the learning process for the controller.
  • Dynamics as a Learnable Correction: Instead of trying to build a perfect humanoid model (which is nearly impossible), it is more effective to treat the model's deficiencies as a learnable part of the control problem.

Key Results

The RFC approach demonstrated significant improvements over the previous state-of-the-art, DeepMimic:

  • Agility: RFC successfully imitated ballet moves that DeepMimic could not reproduce.
  • Convergence: As shown in Figure 2, RFC models converged faster and reached higher final returns across all tested agile motions.
  • Long-term Stability: The dual-policy framework achieved the lowest Mean Angle Error (MAE) and Final Angle Error (FAE) on Human3.6M, outperforming both kinematic and physics-based baselines (see Table 1).
Figure 2: Learning curves of our RFC models and DeepMimic for imitating various agile motions.
Figure 1: Top: A ballet dancer performing highly agile moves like jeté, arabesque and pirouette. Bottom: A humanoid agent controlled by a policy augmented with the proposed residual forces (blue arrows) is able to dance like the performer. The motion is best viewed in our supplementary video.

Limitations and Future Work

The most significant limitation is that external residual forces do not exist in the real world, meaning this method cannot be directly applied to physical robots in its current form. However, the authors suggest RFC could be used as a "warm-up" technique to accelerate learning for real robots or to inform better robot design by identifying where the largest dynamics mismatches occur. Future work could also explore using these learned forces to optimize the physical parameters of the humanoid model itself.

Summary updated

Abstract

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jet\'e. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at https://www.ye-yuan.com/rfc.
BibTeX
@inproceedings{yuan2020residual,
  author = {Ye Yuan and Kris Kitani},
  title = {{Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=AjDoeMBveP},
}
Figure