Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Ye Yuan, Kris Kitani

TL;DR

This paper introduces Residual Force Control (RFC), a novel method to overcome the "dynamics mismatch" in physics-based humanoid control, enabling more realistic imitation of complex human motions like ballet. They also propose a dual-policy framework that uses RFC to synthesize long-term, multi-modal human behaviors without explicit task guidance.

What Problem Does This Paper Address?

Creating realistic virtual humans that can perform complex, agile movements is a long-standing challenge. While reinforcement learning (RL) has made strides in training physics-based humanoid agents to imitate motion capture data, a key hurdle remains: the "dynamics mismatch." This refers to the discrepancy between the physics of a simulated humanoid model and the actual physics of a real human. Consequently, motions that are physically possible for humans might be impossible for the simulated agent, leading to failures in imitation, especially for highly dynamic actions like ballet or for maintaining long-term, coherent movements with complex transitions. Existing methods struggle to bridge this gap, often producing jerky or physically implausible motions.

Methods

The core innovation is Residual Force Control (RFC). Instead of solely relying on the standard control actions (like joint torques) of the humanoid agent, RFC augments the action space with external residual forces. These forces are learned by a separate policy and are applied to the humanoid model during simulation. The idea is that these learned residual forces act as a "correction" to the humanoid's inherent dynamics, effectively compensating for the mismatch and allowing it to perform motions that would otherwise be impossible.

RFC can be implemented in two ways:

  • RFC-Explicit: This method explicitly models residual forces as vectors applied at specific contact points on the humanoid's bodies. This offers interpretability but requires specifying the number and locations of these forces.
  • RFC-Implicit: This approach is more computationally efficient as it directly learns the total residual torques applied to the humanoid's joints, avoiding the need to specify force locations.

To address the challenge of synthesizing long-term, multi-modal human behaviors, the authors propose a dual-policy control framework. This framework combines:

  1. A kinematic policy: This policy, learned using a conditional variational autoencoder (CVAE), predicts future human motions in a kinematic space, capturing multi-modal possibilities and inferring human intent from past motion.
  2. An RFC-based control policy: This policy takes the predicted kinematic motion as a reference and uses RFC to generate physically plausible, real-time control actions for the humanoid agent. This allows for the synthesis of stable, infinite-horizon motions without explicit task definitions.

Datasets

The paper utilizes two primary datasets:

  • CMU Motion Capture (MoCap) database: Used for motion imitation experiments, specifically focusing on eight clips of highly agile motions including ballet (pirouette, arabesque, jeté) and acrobatics (handsprings, backflips, cartwheels, jump kicks, side flips).
  • Human3.6M: A large-scale dataset with millions of frames and multiple subjects, used for evaluating the extended motion synthesis capabilities.
  • EgoMocap: A smaller dataset used for evaluating the extended motion synthesis.

Key Insights and Ideas

  • Dynamics Mismatch is Key: The paper clearly identifies the "dynamics mismatch" as the fundamental bottleneck for imitating complex human motions with physics-based agents.
  • Residual Forces as a "Superpower": RFC effectively grants the humanoid agent a learned "superpower" by allowing it to generate forces that go beyond its inherent physical limitations, enabling it to perform otherwise impossible maneuvers.
  • Bridging Kinematics and Dynamics: The dual-policy framework elegantly combines the predictive power of kinematic models (for forecasting future motion and intent) with the physical realism of dynamics-based control (via RFC), creating a powerful system for long-term motion synthesis.
  • Unsupervised Long-Term Motion Synthesis: The ability to generate diverse, long-term human behaviors without task guidance or user input is a significant step towards more autonomous and versatile virtual characters.

Key Results

  • Improved Motion Imitation: RFC models (both explicit and implicit) consistently outperformed the state-of-the-art DeepMimic method in motion imitation tasks, showing faster convergence and higher quality learned motions across various agile actions.
  • Ballet and Acrobatics Achieved: The proposed RFC approach successfully enabled the humanoid agent to perform highly agile ballet moves (pirouette, arabesque, jeté) and acrobatic feats, which were not reproducible by prior methods.
  • Quantitative Gains in Synthesis: In extended motion synthesis, the RFC-based dual-policy control achieved superior results on Human3.6M and EgoMocap datasets, outperforming kinematics-based baselines (ERD, acLSTM) and another physics-based method (EgoPose) in terms of Mean Angle Error (MAE) and Final Angle Error (FAE). For instance, on Human3.6M (Mix), RFC-Explicit achieved MAE of 2.498 and FAE of 2.893, compared to EgoPose's 0.922 and 1.164 respectively.
  • Ablation Studies Confirm Contributions: Removing either the residual forces or the additive action from the kinematic policy significantly degraded performance, validating the importance of both components.

Limitations and Future Work

  • Simulation Domain Limitation: Currently, RFC is primarily applicable to simulation domains (animation, motion synthesis, pose estimation) because real robots cannot easily generate arbitrary external residual forces.
  • Warm-up Technique Potential: The authors suggest RFC could be used as a "warm-up" technique to accelerate learning for real robots.
  • Agent Design Insights: The learned residual forces could potentially inform and optimize the design of future humanoid robots.
  • Synthesized Video Concerns: The paper acknowledges the broader impact of generating realistic human motion, including the potential for misuse in creating fake videos, and emphasizes the need for research into detection methods.
  • Future Work: Exploring the application of RFC to real robots and further investigating its role in agent design are highlighted as promising future directions.

Summary updated

Abstract

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jeté. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions. Code and videos are available at https://www.ye-yuan.com/rfc.
BibTeX
@inproceedings{yuan2020residual,
  author = {Ye Yuan and Kris Kitani},
  title = {{Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=AjDoeMBveP},
}
Figure