Abstract
We propose a deep learning-based framework that learns data-driven temporal priors to perform 3D human pose estimation from six body-worn Magnetic Inertial Measurement Unit (MIMU) sensors. Our work estimates 3D human pose with associated uncertainty from sparse body-worn sensors. We derive and implement a 3D angle representation that eliminates yaw angle (or magnetometer dependence) and show that 3D human pose is still obtained from this reduced representation, but with enhanced uncertainty. We do not use kinematic acceleration as input and show that it improves the generalization to real sensor data from different subjects as well as accuracy. Our framework is based on a Bi-directional Recurrent Autoencoder (BiRAE). A sliding window is used at inference time, instead of a full sequence (offline mode). The major contribution of our research is that 3D human pose is predicted from sparse sensors with a well-calibrated uncertainty which is correlated with ambiguity and actual errors. We have demonstrated our results on two real sensor datasets, DIP-IMU and Total Capture, achieving state-of-the-art accuracy. Our work confirms that the main limitation of sparse sensor-based 3D human pose prediction is the lack of temporal priors; therefore, fine-tuning on a small synthetic training set of the target domain improves accuracy.
Introduction
Traditional high-fidelity inertial motion capture (i-Mocap) systems typically require 13 to 17 sensors, making them cumbersome and time-consuming to set up for everyday use. While "sparse" systems utilizing only six sensors are significantly more practical, they face two major hurdles: ambiguity and drift. With fewer sensors, many different body poses can appear identical to the system (ambiguity). Furthermore, the magnetometers used to determine heading (yaw) are notoriously sensitive to indoor metal and electronics, leading to significant errors over time (drift). This paper addresses these challenges by learning temporal priors and predicting uncertainty within a robust regression framework.
Methodology
Bi-directional Recurrent Autoencoder (BiRAE)
The core of our system is a Bi-directional Recurrent Autoencoder (BiRAE) based on LSTMs. This architecture is designed to learn the "language" of human movement—the spatial and temporal constraints that make a pose physically plausible. By processing sequences of sensor data, the model uses past and future context to fill in the gaps left by missing sensors.
Magnetometer-Free Mode and Input Simplification
To combat magnetic interference, we derived a 3D angle representation that estimates pose using only pitch and roll. This allows the system to function in a "magnetometer-free" mode, ignoring yaw to avoid drift. Additionally, we found that omitting linear acceleration as an input improved generalization. Acceleration is highly dependent on exact sensor placement and individual body proportions, making it an unreliable feature across different subjects compared to orientation.
Uncertainty Prediction
The model estimates both aleatoric uncertainty (noise in the data) and epistemic uncertainty (limitations of the model's knowledge) using Monte-Carlo Dropout. This allows the system to provide a confidence interval for its predictions, which is critical for safety-sensitive applications like robotics or medical rehabilitation.
Experimental Results and Evaluation
Datasets
- Synthetic Training: A massive dataset based on the SMPL body model and AMASS archive was used to teach the model a wide variety of human motions.
- Real-World Evaluation: The model was tested on two standard benchmarks: DIP-IMU and Total Capture, which provide real sensor readings paired with high-accuracy optical ground truth.
Key Findings
- State-of-the-Art Accuracy: The 6-parameter model achieved a Mean Per Joint Angle Error (MPJAE) of 18.47° on the DIP-IMU dataset after fine-tuning.
- Robustness: In simulations of strong magnetic interference, the magnetometer-free model maintained stable performance, while standard models saw their error rates nearly triple.
- Real-Time Performance: Using a sliding window approach, the system achieves accuracy comparable to offline processing with a latency of only 0.3 seconds, making it suitable for interactive VR/AR applications.
Conclusion
Our research demonstrates that 3D human pose can be accurately predicted from sparse sensors by leveraging temporal priors and robust orientation parameterization. The inclusion of well-calibrated uncertainty bounds provides a diagnostic tool that correlates strongly with actual tracking errors. Future work will focus on expanding the diversity of training data to further bridge the gap between synthetic simulations and "magnetically messy" real-world environments.
Summary updated