Abstract
We present Reusable Motion prior (ReMP), an effective motion prior that can accurately track the temporal evolution of motion in various downstream tasks. Inspired by the success of foundation models, we argue that a robust spatio-temporal motion prior can encapsulate underlying 3D dynamics applicable to various sensor modalities. We learn the rich motion prior from a sequence of complete parametric models of posed human body shape. Our prior can easily estimate poses in missing frames or noisy measurements despite significant occlusion by employing a temporal attention mechanism. More interestingly, our prior can guide the system with incomplete and challenging input measurements to quickly extract critical information to estimate the sequence of poses, significantly improving the training efficiency for mesh sequence recovery. ReMP consistently outperforms the baseline method on diverse and practical 3D motion data, including depth point clouds, LiDAR scans, and IMU sensor data. Project page: https://hojunjang17.github.io/ReMP.
Introduction: A Foundation Model for Human Motion
While "foundation models" have revolutionized natural language processing and computer vision, the field of 3D human motion has historically lacked a unified, robust prior. Capturing high-quality 3D motion data is expensive and difficult to scale, leading to a fragmented landscape where specialized models are built for individual sensor modalities—such as depth cameras, LiDAR, or wearable IMUs. These models often struggle with real-world noise, occlusions, and sparse data because they lack a deep, inherent understanding of human dynamics.
ReMP (Reusable Motion Prior) addresses this by learning the fundamental "grammar" of human movement from massive 3D datasets. By pre-training a transformer-based prior on clean 3D data and then aligning noisy sensor inputs to its latent space, we achieve state-of-the-art accuracy across multiple domains with remarkable data efficiency.
Methodology
The ReMP framework operates in two distinct phases: learning the prior and reusing it for downstream tasks.
1. Learning the Prior
We train a Variational Autoencoder (VAE) with a Transformer backbone on the AMASS dataset, which contains approximately 11,000 clean 3D motion sequences. The model learns to compress sequences of human poses—represented by SMPL parametric model parameters—into a continuous latent space. To instill a deep understanding of temporal dynamics, we employ random temporal masking during training. This forces the model to predict missing frames, effectively teaching it how humans move through time and enabling applications like motion inbetweening.
2. Reusing the Pretrained Prior
Once the prior is established, its weights are frozen. To adapt to a new task (e.g., LiDAR-based pose estimation), we train a lightweight Input Encoder and a Latent Mapper. These components learn to map messy, incomplete sensor data into the pre-trained, clean latent space of the motion prior. This "latent bridge" allows the system to leverage the prior's knowledge of plausible human movement to resolve ambiguities in the input signal.
To ensure robustness, we utilize 6D rotation representations and focus on incremental translations ($\Delta x$) rather than absolute positions. This design choice avoids common pitfalls like coordinate-system ambiguity and dimensional imbalance between rotation and translation data.
Datasets and Evaluation
ReMP was evaluated across a diverse range of synthetic and real-world datasets:
- AMASS: Primary dataset for motion prior training.
- Synthetic CMU: Generated from AMASS for depth-based evaluation.
- B-MHAD (Berkeley MHAD): Real-world depth data from Microsoft Kinect sensors.
- SLOPER4D: A challenging real-world dataset featuring sparse and noisy LiDAR scans in urban environments.
- TotalCapture: Real-world data used for evaluating wearable IMU sensor performance.
Key Results and Insights
ReMP consistently outperformed specialized baselines across all tested modalities, demonstrating the power of a unified motion prior.
- Superior Accuracy: On the CMU depth dataset, ReMP achieved a 4.90° pose error, outperforming the previous state-of-the-art (5.43°). In the sparse SLOPER4D LiDAR environment, it reduced joint error to 21.66cm. For IMU sensors on TotalCapture, it achieved a 12.43° SIP error, surpassing specialized models like PIP.
- Data Efficiency: ReMP trained on only 25% of available data performed as well as or better than baseline models trained on 100% of the data. This suggests that a strong prior drastically reduces the requirement for expensive, labeled sensor data.
- Temporal Context: Unlike frame-wise methods, ReMP's spatio-temporal approach allows it to "see through" occlusions. If a limb is obscured for several frames, the prior reconstructs its position based on the surrounding temporal context.
Conclusion and Future Work
ReMP demonstrates that a robust, reusable motion prior can serve as a foundation for diverse 3D human pose estimation tasks. While currently reliant on the SMPL parametric model and a separate MLP for shape estimation, the framework provides a scalable path forward for human motion analysis. Future research will explore extending this approach to multi-person interactions and complex human-object manipulations, where temporal constraints are even more vital for resolving physical ambiguities.
Summary updated