Perpetual Humanoid Control for Real-time Simulated Avatars

Zhengyi Luo, Jinkun Cao, Alexander Winkler, Kris Kitani, Weipeng Xu

Abstract

Physically simulated humanoids offer a path toward realistic and interactive virtual characters, yet they are notoriously difficult to control. We present Perpetual Humanoid Control (PHC), a system that enables avatars to mimic complex human motions in real-time without requiring resets. By employing a Progressive Multiplicative Control Policy (PMCP), the system scales to large-scale motion datasets and naturally recovers from falls or noisy input. PHC eliminates the need for external "Residual Force Control," resulting in physically plausible movements that are robust enough for live applications.

Introduction

The control of physically simulated humanoids is a long-standing challenge in computer animation and robotics. Unlike kinematic animation, simulated characters must contend with gravity, momentum, and balance. Traditional approaches often fail when faced with three primary issues: the Reset Problem, where characters must teleport back to a start pose after a fall; the "Hand of God", or the use of unphysical external forces (Residual Force Control) to maintain balance; and Catastrophic Forgetting, where a model loses proficiency in simple tasks while learning complex ones.

PHC demonstrates high-fidelity tracking across a wide variety of complex human motions without external stabilizing forces.

Methods

The core of PHC is the Progressive Multiplicative Control Policy (PMCP). This architecture adopts a "divide and conquer" strategy to manage the complexity of human motion.

Progressive Learning and Composition

Instead of a single monolithic network, PMCP utilizes a series of "primitives." The system begins with a small network trained on simple motions. As it encounters more difficult sequences, it freezes the existing weights and adds a new primitive to handle the failure cases. A high-level Composer network then learns to blend these primitives in real-time, ensuring smooth transitions between diverse actions.

Stability and Recovery

To ensure "perpetual" operation, the authors introduced two key mechanisms:

  • Fail-state Recovery: A dedicated primitive is trained specifically for the "getting up" motion. If the character falls or deviates significantly from the reference, this module restores the character to its feet.
  • Relaxed Early Termination (RET): During training, standard methods stop the simulation if any joint deviates from the reference. RET allows for slight deviations in the extremities (ankles and toes) to prioritize overall balance, significantly enhancing stability.

Key Insights

A significant finding of this work is that 3D keypoints are sufficient for high-fidelity imitation, removing the need for complex joint rotation data. This allows the system to be driven by standard webcam-based pose estimators. Furthermore, the physics simulation acts as a natural regularizer, cleaning up the jitter often found in AI-generated pose data.

Results

PHC demonstrates state-of-the-art performance across several benchmarks. It successfully imitated 98.9% of the AMASS dataset—comprising over 40 hours of motion—without any external stabilizing forces.

The PHC system enables real-time tracking of human motion from standard video input, making it suitable for live VR and AR applications.

In evaluations using the Human3.6M dataset, PHC maintained a ~90% success rate on noisy, real-world video data. In contrast, previous methods often saw success rates drop below 20% when their external forces were disabled. The system is highly efficient, running at over 30 FPS on consumer-grade GPUs, and the recovery module provides nearly 100% reliability in returning a fallen character to a tracking state within seconds.

Conclusion and Future Work

Perpetual Humanoid Control provides a robust framework for real-time, physics-based animation. While the system currently struggles with extreme dynamic motions like backflips and lacks environmental awareness (e.g., navigating terrain or interacting with objects), it sets a new standard for stability and scale. Future research will focus on integrating terrain perception and closing the loop between vision and physics to further improve naturalness and responsiveness.

Summary updated

Abstract

We present a physics-based humanoid controller that achieves high-fidelity motion imitation and fault-tolerant behavior in the presence of noisy input (e.g. pose estimates from video or generated from language) and unexpected falls. Our controller scales up to learning ten thousand motion clips without using any external stabilizing forces and learns to naturally recover from fail-state. Given reference motion, our controller can perpetually control simulated avatars without requiring resets. At its core, we propose the progressive multiplicative control policy (PMCP), which dynamically allocates new network capacity to learn harder and harder motion sequences. PMCP allows efficient scaling for learning from large-scale motion databases and adding new tasks, such as fail-state recovery, without catastrophic forgetting. We demonstrate the effectiveness of our controller by using it to imitate noisy poses from video-based pose estimators and language-based motion generators in a live and real-time multi-person avatar use case.
BibTeX
@inproceedings{luo2023perpetual,
  author = {Zhengyi Luo and Jinkun Cao and Alexander Winkler and Kris Kitani and Weipeng Xu},
  title = {{Perpetual Humanoid Control for Real-time Simulated Avatars}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01000},
}
Figure