MotionBERT: Unified Pretraining for Human Motion Analysis

Wentao Zhu, Xiaoxuan Ma, Zhaoyang Liu, Libin Liu, Wayne Wu, Yizhou Wang

Abstract

We present MotionBERT, a unified pretraining framework, to tackle different sub-tasks of human motion analysis including 3D pose estimation, skeleton-based action recognition, and mesh recovery. The proposed framework is capable of utilizing all kinds of human motion data resources, including motion capture data and in-the-wild videos. During pretraining, the pretext task requires the motion encoder to recover the underlying 3D motion from noisy partial 2D observations. The pretrained motion representation thus acquires geometric, kinematic, and physical knowledge about human motion and therefore can be easily transferred to multiple downstream tasks. We implement the motion encoder with a novel Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. More importantly, the proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with 1-2 linear layers, which demonstrates the versatility of the learned motion representations.

Introduction

Human motion analysis encompasses a variety of critical tasks, including 3D pose estimation, skeleton-based action recognition, and human mesh recovery. Traditionally, these tasks have been addressed using specialized, isolated models. This fragmented approach faces two primary challenges: data heterogeneity and redundant learning. High-fidelity 3D Motion Capture (Mocap) data is often restricted to indoor environments, while diverse "in-the-wild" videos lack 3D ground truth. Furthermore, models for different tasks often independently learn similar kinematic constraints that could ideally be shared.

MotionBERT addresses these challenges by introducing a unified pretraining framework. By learning a versatile motion representation from heterogeneous data sources, MotionBERT can be effectively transferred to multiple downstream tasks with minimal task-specific adjustments.

Methodology

The MotionBERT framework operates in two distinct stages: unified pretraining and task-specific finetuning.

Overview of the MotionBERT framework, illustrating the two-stage process of unified pretraining and task-specific finetuning.

Unified Pretraining via 2D-to-3D Lifting

The core of the pretraining phase is a "cloze" pretext task. The model is presented with 2D skeleton sequences that have been corrupted with random noise and masking (hiding certain joints or frames). The objective is to reconstruct the original 3D motion. This process compels the motion encoder to internalize geometric constraints, skeletal hierarchies, and temporal dynamics. By using 2D skeletons as a universal input format, the model can leverage both projected 3D Mocap data and 2D poses extracted from unannotated in-the-wild videos, effectively solving the data heterogeneity problem.

Dual-stream Spatio-temporal Transformer (DSTformer)

To effectively capture the complexities of human motion, the authors propose the DSTformer. Unlike standard transformers that may struggle to balance spatial and temporal relationships, the DSTformer employs a dual-stream architecture.

Detailed architecture of the DSTformer, featuring parallel spatial-to-temporal and temporal-to-spatial streams with adaptive fusion.

One stream processes spatial joint relationships before temporal dynamics, while the other reverses this order. These streams are fused adaptively, allowing the model to dynamically weight spatial or temporal cues based on the specific motion sequence it is processing.

Task-specific Finetuning

Following pretraining, the motion encoder is transferred to downstream tasks. A simple head, typically consisting of one or two linear layers, is appended to the encoder to perform specific tasks such as action classification or mesh regression. This minimal overhead demonstrates the power of the learned representations.

Experiments and Results

The model was trained on a massive collection of datasets, including Human3.6M and AMASS for 3D motion, PoseTrack and InstaVariety for in-the-wild 2D data, and NTU-RGB+D for action recognition.

Performance Benchmarks

MotionBERT achieves state-of-the-art (SOTA) results across all evaluated tasks:

  • 3D Pose Estimation: Achieved a record-low error of 37.5mm MPJPE on the Human3.6M benchmark, surpassing specialized SOTA models.
  • Action Recognition: Attained 93.0% accuracy on NTU-RGB+D (X-Sub). It demonstrated exceptional performance in one-shot action recognition, reaching 67.4% accuracy—a significant improvement over previous methods.
  • Mesh Recovery: When integrated as a refiner for existing methods, MotionBERT significantly reduced errors and enhanced temporal smoothness on benchmarks like 3DPW and COCO.

Conclusion

MotionBERT demonstrates that a unified pretraining approach can yield highly versatile and robust representations for human motion analysis. By bridging the gap between different data types and tasks, it sets a new standard for the field. Future research will explore extending the framework to multi-person interactions and incorporating environmental visual context to further enhance complex action understanding.

Summary updated

Abstract

We present MotionBERT, a unified pretraining framework, to tackle different sub-tasks of human motion analysis including 3D pose estimation, skeleton-based action recognition, and mesh recovery. The proposed framework is capable of utilizing all kinds of human motion data resources, including motion capture data and in-the-wild videos. During pretraining, the pretext task requires the motion encoder to recover the underlying 3D motion from noisy partial 2D observations. The pretrained motion representation thus acquires geometric, kinematic, and physical knowledge about human motion and therefore can be easily transferred to multiple downstream tasks. We implement the motion encoder with a novel Dual-stream Spatio-temporal Transformer (DSTformer) neural network. It could capture long-range spatio-temporal relationships among the skeletal joints comprehensively and adaptively, exemplified by the lowest 3D pose estimation error so far when trained from scratch. More importantly, the proposed framework achieves state-of-the-art performance on all three downstream tasks by simply finetuning the pretrained motion encoder with 1-2 linear layers, which demonstrates the versatility of the learned motion representations.
BibTeX
@misc{zhu2022motionbert,
  author = {Wentao Zhu and Xiaoxuan Ma and Zhaoyang Liu and Libin Liu and Wayne Wu and Yizhou Wang},
  title = {{MotionBERT: Unified Pretraining for Human Motion Analysis}},
  year = {2022},
  eprint = {2210.06551},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.06551},
}
Figure