Abstract
We propose a novel deep generative model based on causal convolutions for multi-subject motion modeling and synthesis, which is inspired by the success of WaveNet in multi-subject speech synthesis. However, it is nontrivial to adapt WaveNet to handle high-dimensional and physically constrained motion data. To this end, we add an encoder and a decoder to the WaveNet to translate the motion data into features and back to the predicted motions. We also add 1D convolution layers to take skeleton configuration as an input to model skeleton variations across different subjects. As a result, our network can scale up well to large-scale motion data sets across multiple subjects and support various applications, such as random and controllable motion synthesis, motion denoising, and motion completion, in a unified way. Complex motions, such as punching, kicking and, kicking while punching, are also well handled. Moreover, our network can synthesize motions for novel skeletons not in the training dataset. After fine-tuning the network with a few motion data of the novel skeleton, it is able to capture the personalized style implied in the motion and generate high-quality motions for the skeleton. Thus, it has the potential to be used as a pre-trained network in few-shot learning for motion modeling and synthesis. Experimental results show that our model can effectively handle the variation of skeleton configurations, and it runs fast to synthesize different types of motions on-line. We also perform user studies to verify that the quality of motions generated by our network is superior to the motions of state-of-the-art human motion synthesis methods.
Introduction
Generating realistic human motion is a classic challenge in computer graphics and artificial intelligence. Human movement is governed by highly nonlinear kinematic systems, and every individual possesses a unique "style"—a tall person walks differently than a short person, and a specific action like a "zombie walk" varies significantly based on an actor's proportions.
Existing models often struggle with three primary issues:
- Scaling: Handling datasets with many different subjects without losing the nuances of individual styles.
- Stability: Autoregressive models like RNNs often suffer from error accumulation, where small mistakes in one frame lead to catastrophic failures in later frames.
- Generalization: Most models are locked to the specific skeleton used during training; animating a character with different proportions often results in artifacts like "foot sliding."
To address these, we introduce CCNet, a causal convolutional neural network that treats skeleton configurations as direct inputs, allowing it to model diverse movement styles across multiple subjects and generalize to entirely new skeletons.
Methodology
CCNet leverages 1D Causal Convolutions to ensure that predictions for the next frame depend only on previous frames, preserving temporal consistency. The architecture is designed to be skeleton-aware, injecting physical proportions into the learning process.
Key Technical Components
- Dilated Convolutions: By skipping steps in convolutional layers, the network achieves a massive receptive field (41 frames) without an excessive parameter count.
- Skeleton-Aware Residual Blocks: Skeleton configurations (height and joint positions) are injected into every residual block. This constant "reminder" of the character's physical proportions allows the model to learn the physics-based relationship between body size and movement.
- Probabilistic Modeling: Instead of a deterministic next pose, CCNet outputs a Probability Density Function (PDF), enabling natural variation and robustness against noisy inputs.
- Multi-Term Loss Function: Training is guided by a Gaussian loss for accuracy, a smoothness loss to prevent jitter, and a specific foot-contact loss to assist Inverse Kinematics (IK) systems in preventing floor penetration.
Datasets and Multi-Subject Modeling
The model was trained and validated using three primary sources:
- A New Large-Scale Dataset: Recorded 12 subjects (9 men, 3 women) performing 10 motion types, totaling 486,282 frames, including transitional clips.
- CMU Mocap Dataset: Used as a standard industry benchmark for cross-validation.
- Human3.6M (H3.6M): Utilized to test motion prediction capabilities against state-of-the-art architectures.
Experimental Results
CCNet demonstrates significant improvements over traditional Recurrent Neural Networks (RNNs), which are often harder to train and prone to vanishing gradient problems.
- Trajectory Following: In path-following tests, CCNet achieved a mean error of 27.8 cm, vastly outperforming the Phase-Functioned Neural Network (PFNN) error of 158.6 cm.
- Denoising and Robustness: The model proved significantly more robust at cleaning noisy motion capture data compared to baselines like ERD-4LR and DAE-LSTM.
- Real-Time Performance: Running at approximately 65 FPS, the model is suitable for interactive applications like video games or VR.
- Few-Shot Learning: One of the most impressive findings is the power of fine-tuning. By training on a small amount of data for a "novel" skeleton, the pre-trained CCNet can synthesize high-quality, personalized motion that respects new character proportions.
Discussion and Future Work
While CCNet is a powerful tool for motion synthesis, it currently requires a separate Inverse Kinematics (IK) pass to completely eliminate foot sliding. Additionally, the model assumes a consistent topology (number and connection of joints), even as bone lengths change. Future research will explore extending the model to handle entirely different skeleton structures, such as quadrupeds or characters with non-humanoid limb configurations.
Summary updated