MotionAug: Augmentation with Physical Correction for Human Motion Prediction

Takahiro Maeda, Norimichi Ukita

Abstract

Predicting human motion is critical for autonomous systems, yet high-quality datasets are scarce. We present MotionAug, an augmentation framework that combines generative modeling with physical correction. MotionAug utilizes a modified Variational AutoEncoder (VAE) and Inverse Kinematics (IK) to synthesize candidate motions, which are then refined through a physics simulator to ensure physical plausibility. To address simulation bias and computational overhead, we introduce a PD-residual force mechanism and a debiasing step. Experimental results on the HDM05 dataset demonstrate that MotionAug significantly enhances the performance of motion prediction models, particularly in data-constrained scenarios.

Introduction

Deep learning models for human motion prediction require vast amounts of high-quality data. Traditional data augmentation techniques, such as adding noise or simple geometric transformations, often result in physically implausible motions—such as "floating" characters or limbs penetrating the floor. These artifacts degrade model performance. MotionAug addresses this by creating a pipeline that synthesizes diverse motions and corrects them using the laws of physics.

Methodology

The MotionAug framework consists of three primary stages: motion synthesis, physical correction, and debiasing.

Overview of the MotionAug pipeline, showing the transition from sparse motion to physically plausible dense motions through synthesis and correction stages.

1. Motion Synthesis

The synthesis stage employs two methods to generate diverse motion candidates:

  • Modified VAE: A Variational AutoEncoder trained with an adversarial loss to maintain motion sharpness. To handle small datasets, a "sampling-near-samples" strategy is used, where new motions are sampled from latent space regions close to clusters of real training data, avoiding "unlearned" areas that produce unrealistic poses.
  • Semi-Automatic IK: Users define target spaces for specific joints (e.g., a foot's landing zone), and the system automatically generates valid Inverse Kinematics-based sequences.

2. Motion Correction

Synthesized motions often contain physical artifacts like "footskating." MotionAug uses imitation learning within the Bullet Physics simulator to rectify these. A simulated character attempts to mimic the synthesized motion, and the simulator's physics engine naturally filters out impossible movements. To accelerate this computationally expensive process, a PD-residual force mechanism is introduced, reducing training time from 30 hours to approximately 9 hours by narrowing the learning search space.

3. Motion Debiasing

Physics simulators typically use rigid-body approximations that do not perfectly capture human soft-tissue dynamics. This creates a "dynamic mismatch." MotionAug employs a neural network to map simulated motions back to the distribution of real human movement, ensuring the augmented data remains relevant for real-world applications.

Experimental Results

The framework was evaluated using the HDM05 Motion Database, which features dynamic actions like punching and kicking. Testing across RNN, GCN, and Transformer architectures showed consistent improvements.

Qualitative comparison of motion samples. The proposed VAE generates more dynamic and realistic poses compared to GAN and vanilla VAE baselines.

As shown in the qualitative comparison, the proposed VAE avoids the "mode collapse" seen in GANs and the over-smoothed, static poses typical of vanilla VAEs when data is scarce. Quantitatively, MotionAug reduced prediction error significantly; for instance, in RNN-based "kick" motion prediction, error at 400ms dropped from 2.45 to 1.80 radians.

Conclusion

MotionAug effectively bridges the gap between generative data augmentation and physical realism. By utilizing physics as a regularizer and introducing efficient correction mechanisms, the framework provides a robust solution for training motion prediction models on limited datasets. Future work will focus on scaling the correction phase via GPU acceleration and extending the method to 2D video inputs.

Summary updated

Abstract

This paper presents a motion data augmentation scheme incorporating motion synthesis encouraging diversity and motion correction imposing physical plausibility. This motion synthesis consists of our modified Variational AutoEncoder (VAE) and Inverse Kinematics (IK). In this VAE, our proposed sampling-near-samples method generates various valid motions even with insufficient training motion data. Our IK-based motion synthesis method allows us to generate a variety of motions semi-automatically. Since these two schemes generate unrealistic artifacts in the synthesized motions, our motion correction rectifies them. This motion correction scheme consists of imitation learning with physics simulation and subsequent motion debiasing. For this imitation learning, we propose the PD-residual force that significantly accelerates the training process. Furthermore, our motion debiasing successfully offsets the motion bias induced by imitation learning to maximize the effect of augmentation. As a result, our method outperforms previous noise-based motion augmentation methods by a large margin on both Recurrent Neural Network-based and Graph Convolutional Network-based human motion prediction models. The code is available at https://github.com/meaten/MotionAug.
BibTeX
@inproceedings{maeda2022motionaug,
  author = {Takahiro Maeda and Norimichi Ukita},
  title = {{MotionAug: Augmentation with Physical Correction for Human Motion Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2022},
  doi = {10.1109/cvpr52688.2022.00632},
}
Figure