Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening

Yunhao Li, Zhenbo Yu, Yucheng Zhu, Bingbing Ni, Guangtao Zhai, Wei Shen

Abstract

Human motion synthesis is a long-standing problem with various applications in digital twins and the Metaverse. However, modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions and consequently they usually produce unrealistic human motions. In order to solve this problem, we propose a system "Skeleton2Humanoid" which performs physics-oriented motion correction at test time by regularizing synthesized skeleton motions in a physics simulator. Concretely, our system consists of three sequential stages: (I) test time motion synthesis network adaptation, (II) skeleton to humanoid matching and (III) motion imitation based on reinforcement learning (RL). Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations. Stage II performs an analytical inverse kinematics strategy, which converts the optimized human skeleton motions to humanoid robot motions in a physics simulator, then the converted humanoid robot motions can be served as reference motions for the RL policy to imitate. Stage III introduces a curriculum residual force control policy, which drives the humanoid robot to mimic complex converted reference motions in accordance with the physical law. We verify our system on a typical human motion synthesis task, motion-in-betweening. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy. Code will be released for research purposes at: https://github.com/michaelliyunhao/Skeleton2Humanoid

Introduction

In 3D animation, "motion in-betweening" is the task of generating smooth transitions between distant keyframes. While deep learning models have become proficient at predicting joint trajectories, they often lack an understanding of physical constraints. This "physics-blindness" leads to common artifacts such as foot sliding, ground penetration, and jerky, unnatural transitions. These issues significantly degrade the realism required for high-fidelity applications like digital twins and the Metaverse.

Skeleton2Humanoid addresses these challenges by bridging the gap between mathematical kinematic predictions and physical reality. By integrating a physics simulator into the animation pipeline, the system ensures that characters not only reach their target poses but do so while obeying the laws of gravity, friction, and joint limits.

The Skeleton2Humanoid Framework

The proposed system utilizes a three-stage pipeline to transform raw kinematic predictions into physically plausible humanoid motions.

Stage I: Test Time Adaptation (TTA)

Standard pre-trained models often suffer from domain shift when applied to new, unseen sequences. To mitigate this, Stage I introduces a Test Time Adaptation strategy. The system refines the skeleton motion on the fly using self-supervised losses. Specifically, it employs a foot contact consistency loss to prevent sliding and a motion smoothness loss to eliminate jitter. This stage ensures the skeleton motion is as clean as possible before entering the physical simulation.

Synthesized human skeleton motion representing the kinematic output of Stage I.

Stage II: Skeleton to Humanoid Matching

To transition from a mathematical skeleton to a physical entity, the system employs an analytical Inverse Kinematics (IK) method. This method maps 3D joint positions to the Euler angles required by a humanoid model in the MuJoCo physics simulator. This mapping is designed to be general, allowing it to handle differences in bone structure between the source skeleton and the target humanoid character.

Stage III: Motion Imitation via Reinforcement Learning

The final stage involves training a reinforcement learning (RL) policy to drive the humanoid character to mimic the reference motion generated in Stage II. The authors introduce a Curriculum Residual Force Control (CRP) policy. This approach uses "curriculum forces"—external helper forces that initially assist the character in maintaining balance. As training progresses, these forces are gradually phased out, forcing the RL agent to learn natural balancing and movement strategies that rely solely on internal joint torques.

Physically-simulated humanoid character mimicking the reference motion in Stage III.

Experimental Evaluation

The system was evaluated on the LaFAN1 dataset, a high-quality motion capture benchmark. For physics-based testing, the authors curated a "Noobjects LaFAN1" subset, removing sequences involving complex human-object interactions (like climbing) where ground-truth geometry for the objects was unavailable.

Performance was measured using both accuracy metrics (L2Q, L2P) and physical plausibility metrics:

  • Foot Penetration (FP): Measures how much the character's feet clip through the floor.
  • Jerkiness (JQ): Quantifies the smoothness of the motion.

Results and Discussion

Skeleton2Humanoid demonstrated significant improvements over baseline kinematic models.

  • Physical Plausibility: The RL-based imitation stage effectively eliminated foot sliding and ground penetration, producing motions that look significantly more realistic.
  • Accuracy: Surprisingly, the Test Time Adaptation in Stage I not only improved plausibility but also set a new state-of-the-art for accuracy on the LaFAN1 benchmark. This suggests that physical constraints can serve as a powerful regularizer for kinematic accuracy.
  • Efficiency: The analytical IK mapping is highly efficient, processing sequences in approximately 0.122 seconds, making it suitable for integration into modern animation workflows.

Conclusion

Skeleton2Humanoid provides a robust framework for generating physically realistic human animations. By combining the predictive power of deep learning with the rigorous constraints of a physics simulator, it solves persistent issues like foot sliding and ground penetration. While currently limited by the need for precise 3D geometry in human-object interactions, future work aims to extend this system to more complex environments and multi-character interactions.

Summary updated

Abstract

Human motion synthesis is a long-standing problem with various applications in digital twins and the Metaverse. However, modern deep learning based motion synthesis approaches barely consider the physical plausibility of synthesized motions and consequently they usually produce unrealistic human motions. In order to solve this problem, we propose a system ``Skeleton2Humanoid'' which performs physics-oriented motion correction at test time by regularizing synthesized skeleton motions in a physics simulator. Concretely, our system consists of three sequential stages: (I) test time motion synthesis network adaptation, (II) skeleton to humanoid matching and (III) motion imitation based on reinforcement learning (RL). Stage I introduces a test time adaptation strategy, which improves the physical plausibility of synthesized human skeleton motions by optimizing skeleton joint locations. Stage II performs an analytical inverse kinematics strategy, which converts the optimized human skeleton motions to humanoid robot motions in a physics simulator, then the converted humanoid robot motions can be served as reference motions for the RL policy to imitate. Stage III introduces a curriculum residual force control policy, which drives the humanoid robot to mimic complex converted reference motions in accordance with the physical law. We verify our system on a typical human motion synthesis task, motion-in-betweening. Experiments on the challenging LaFAN1 dataset show our system can outperform prior methods significantly in terms of both physical plausibility and accuracy. Code will be released for research purposes at: https://github.com/michaelliyunhao/Skeleton2Humanoid
BibTeX
@inproceedings{li2022skeleton,
  author = {Yunhao Li and Zhenbo Yu and Yucheng Zhu and Bingbing Ni and Guangtao Zhai and Wei Shen},
  title = {{Skeleton2Humanoid: Animating Simulated Characters for Physically-plausible Motion In-betweening}},
  booktitle = {ACM Multimedia},
  year = {2022},
  doi = {10.1145/3503161.3548093},
}
Figure