Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis

Ye Yuan, Kris M. Kitani

Introduction

Simulating realistic human motion in physics-based environments remains a significant challenge in computer graphics and robotics. While kinematic methods can produce visually appealing results, they often suffer from artifacts like "foot sliding" or "ghosting." Physics-based reinforcement learning (RL) offers a more grounded alternative, yet these models frequently struggle to replicate highly agile human behaviors—such as ballet pirouettes or acrobatic flips—due to "dynamics mismatch." This mismatch occurs because simplified simulated humanoids lack the biomechanical complexity of real humans, making certain captured motions physically impossible to execute within the simulator's constraints.

To address this, we introduce Residual Force Control (RFC), a technique that augments a humanoid's internal joint torques with external "residual forces." This approach enables virtual characters to imitate complex, high-agility motions that were previously unattainable for standard physics-based controllers.

Figure 1: Comparison between real-world human motion and the physics-based humanoid imitation using RFC.

Methodology

The core innovation of this work is the integration of external residual forces into the agent's action space. These forces act as "invisible helping hands" that provide the necessary pushes or pulls to compensate for simulation inaccuracies.

Residual Force Control (RFC)

RFC allows the RL agent to apply external forces to maintain balance and accuracy. To ensure the resulting motion remains physically plausible, a heavy penalty is applied to the use of these forces, forcing the agent to rely on internal joint torques whenever possible. The paper explores two variants:

  • RFC-Explicit: Models forces applied to specific body parts, such as the feet.
  • RFC-Implicit: A more computationally efficient version that applies a general torque to the character's root.

Dual-Policy Framework

For long-term and diverse motion synthesis, a two-stage pipeline is employed:

  1. Kinematic Policy: A Conditional Variational Autoencoder (CVAE) generates physically plausible future poses based on historical data.
  2. RFC Policy: A low-level controller executes these predicted poses within the physics engine, utilizing residual forces to bridge the gap between kinematic "dreams" and rigid-body physics.

Results and Evaluation

The framework was evaluated across three major benchmarks: CMU MoCap (agile motions), Human3.6M (long-term synthesis), and EgoMocap (head-mounted camera data).

Figure 2: Training performance (Return vs. Epoch) across various agile tasks. RFC variants significantly outperform the DeepMimic baseline in both convergence speed and final return.

As shown in Figure 2, RFC-based methods demonstrate superior learning efficiency and higher final performance compared to state-of-the-art baselines like DeepMimic. The addition of residual forces provides a smoother gradient for the RL agent, leading to faster convergence even in complex tasks like ballet and acrobatics.

Quantitatively, the RFC-based dual policy achieved significantly lower Mean Angle Error (MAE) and Final Angle Error (FAE) on the Human3.6M dataset compared to prior physics-based and kinematic methods. Furthermore, the RFC-Implicit model is highly efficient, adding only ~0.4ms of computational overhead per step, making it suitable for real-time applications.

Discussion and Future Work

RFC serves as a vital "invisible bridge" between kinematic data and rigid-body physics. By acknowledging the inherent imperfections of simulation models, it allows for much higher visual fidelity in motion imitation. Moreover, the framework enables task-free synthesis, allowing characters to generate "infinite-horizon" natural behaviors without explicit goal guidance.

While RFC cannot be directly deployed on physical hardware due to its reliance on external forces, it offers a powerful tool for simulation-based training. Future work may involve using RFC as a "warm-up" phase, gradually decaying the residual forces to zero to facilitate real-world transfer. Additionally, the magnitude and location of the required residual forces could inform the design of more capable physical robots by highlighting areas where better weight distribution or higher torque is needed.

Summary updated

Abstract

Reinforcement learning has shown great promise for synthesizing realistic human behaviors by learning humanoid control policies from motion capture data. However, it is still very challenging to reproduce sophisticated human skills like ballet dance, or to stably imitate long-term human behaviors with complex transitions. The main difficulty lies in the dynamics mismatch between the humanoid model and real humans. That is, motions of real humans may not be physically possible for the humanoid model. To overcome the dynamics mismatch, we propose a novel approach, residual force control (RFC), that augments a humanoid control policy by adding external residual forces into the action space. During training, the RFC-based policy learns to apply residual forces to the humanoid to compensate for the dynamics mismatch and better imitate the reference motion. Experiments on a wide range of dynamic motions demonstrate that our approach outperforms state-of-the-art methods in terms of convergence speed and the quality of learned motions. Notably, we showcase a physics-based virtual character empowered by RFC that can perform highly agile ballet dance moves such as pirouette, arabesque and jeté. Furthermore, we propose a dual-policy control framework, where a kinematic policy and an RFC-based policy work in tandem to synthesize multi-modal infinite-horizon human motions without any task guidance or user input. Our approach is the first humanoid control method that successfully learns from a large-scale human motion dataset (Human3.6M) and generates diverse long-term motions.
BibTeX
@inproceedings{yuan2020residual,
  author = {Ye Yuan and Kris M. Kitani},
  title = {{Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=AjDoeMBveP},
}
Figure