TL;DR
Skeleton2Humanoid is a new three-stage framework that transforms "floaty" AI-generated animations into physically realistic motions. By combining test-time optimization, analytical inverse kinematics, and reinforcement learning, it ensures that characters don't just move accurately, but also obey the laws of physics—no more feet sliding through floors or jerky, impossible joint rotations.
What Problem Does This Paper Address?
In the world of 3D animation, "motion in-betweening" is the task of filling in the frames between two distant keyframes (e.g., a character starting a run and then finishing a jump). While modern deep learning models are great at predicting where joints should be, they are "physics-blind."
The result? Synthesized characters often suffer from "foot sliding" (skating across the floor), "ground penetration" (limbs clipping through the terrain), and "jerky transitions" that look robotic. These artifacts break immersion in digital twins and the Metaverse. This paper tackles the challenge of making these motions "physically plausible"—ensuring they look like they were performed by a real body subject to gravity and friction.
Methods
The authors propose a sequential three-stage pipeline:
- Stage I: Test Time Adaptation (TTA): Instead of just accepting what a pre-trained network spits out, the system refines the motion on the fly. It uses two self-supervised losses—foot contact consistency (forcing feet to stay still when touching the ground) and motion smoothness—to "correct" the skeleton before it ever hits the simulator.
- Stage II: Skeleton to Humanoid Matching: To move from a mathematical skeleton to a physical robot, the authors developed a general analytical Inverse Kinematics (IK) method. This maps 3D joint positions to the specific Euler angles required by a humanoid character, even if the skeleton and the humanoid have different bone structures.
- Stage III: Motion Imitation via Reinforcement Learning (RL): Finally, the character is placed in the MuJoCo physics simulator. An RL policy, called Curriculum Residual Force Control (CRP), learns to drive the character to mimic the reference motion. It uses a "curriculum" approach, starting with high external "helper" forces to keep the character upright and gradually removing them as the character learns to balance and move naturally.
Datasets
- LaFAN1: A large-scale, high-quality motion capture dataset featuring complex actions like fighting, crawling, and dancing.
- Noobjects LaFAN1: A specific subset created by the authors. Since physics simulators require 3D models of every object a character touches, the authors removed sequences involving climbing or interacting with obstacles (where 3D ground truth for the objects was missing) to focus purely on ground-based physical plausibility.
Key Insights and Ideas
- The Domain Shift Problem: The authors realized that even the best models fail on test data because of "domain shift." TTA acts as a bridge, allowing the model to adapt its predictions to the specific nuances of a new motion sequence at inference time.
- Physics as a Regularizer: Rather than trying to bake physics into the neural network architecture (which is incredibly hard), they use a physics simulator as the final "judge" and "corrector" of the motion.
- Curriculum Forces: Training an RL agent to mimic a complex backflip from scratch is nearly impossible. By starting with "training wheels" (residual forces) and slowly taking them off, the agent achieves much higher performance and stability.
Key Results
- Physical Plausibility: The system significantly outperformed baselines in physics-based metrics. It achieved much lower Foot Penetration (FP) and Jerkiness (JQ) compared to standard kinematic models.
- Accuracy: On the LaFAN1 benchmark, the Test Time Adaptation stage achieved a new state-of-the-art for accuracy (measured by L2Q and L2P metrics), proving that "looking real" and "being accurate" can go hand-in-hand.
- Efficiency: The analytical IK method is remarkably fast, processing a sequence in roughly 0.122 seconds, making it viable for practical animation pipelines.
Limitations and Future Work
The primary limitation is Human-Object Interaction. Because the physics simulator requires precise 3D collision geometry for every object in a scene, the current system struggles with motions like climbing a ladder or sitting in a chair if those objects aren't perfectly modeled.
The authors suggest that future work will focus on expanding the system to handle these complex interactions and even human-to-human physical contact, which remains a "holy grail" for simulated character animation.
Summary updated