Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time

Yinghao Huang, Manuel Kaufmann, Emre Aksan, Michael J. Black, Otmar Hilliges, Gerard Pons-Moll

TL;DR

This paper introduces "Deep Inertial Poser" (DIP), a novel deep learning method that reconstructs full-body human pose in real-time using only six wearable Inertial Measurement Units (IMUs). This breakthrough overcomes the limitations of traditional motion capture systems, enabling accurate and non-intrusive pose tracking in everyday environments.

What Problem Does This Paper Address?

Capturing accurate 3D human pose is crucial for applications like gaming, virtual/augmented reality, and biomechanical analysis. However, existing methods face significant challenges:

  • Infrastructure Dependency: Traditional motion capture systems (e.g., Vicon) require expensive, specialized setups with multiple cameras and markers.
  • Line-of-Sight Issues: Camera-based methods struggle with occlusions, especially in scenarios like sitting at a desk or outdoor environments.
  • Intrusiveness: Even marker-less camera approaches can be computationally demanding or require controlled environments.
  • Under-constrained Problem: Estimating full 3D pose from a sparse set of IMUs is inherently difficult because multiple poses can produce the same sensor readings.
  • Real-time Performance: Previous methods that achieved good accuracy with sparse IMUs relied on computationally expensive offline optimization, making real-time application infeasible.

DIP aims to solve these problems by providing a real-time, minimally invasive, and infrastructure-free solution for 3D human pose reconstruction.

Methods

DIP employs a deep learning approach, specifically a recurrent neural network (RNN) architecture, to learn the complex mapping from sparse IMU data to human pose parameters.

  • SMPL Model: The authors utilize the Skinned Multi-Person Linear Model (SMPL) as a statistical prior for human shape and pose. This model is used both to generate synthetic training data and as the target output for the network.
  • Synthetic Data Generation: To overcome the scarcity of large-scale, synchronized IMU and ground-truth pose datasets, DIP synthesizes IMU data from existing motion capture datasets (like AMASS). Virtual IMUs are placed on the SMPL mesh, and their orientations and accelerations are derived using forward kinematics and finite differences.
  • Bi-directional RNN (BiRNN) with LSTM: To capture temporal dependencies in motion, DIP uses a BiRNN with Long Short-Term Memory (LSTM) cells. This architecture allows the network to leverage information from both past and future frames during training, leading to smoother and more accurate predictions. At test time, a sliding window approach is used to maintain real-time performance.
  • Auxiliary Acceleration Reconstruction Loss: To address the challenge of accurately modeling acceleration data (which can be noisy and difficult to synthesize), an auxiliary loss function is introduced. This forces the network to reconstruct input accelerations in addition to pose, improving the propagation of acceleration information and leading to better pose predictions.
  • Dropout Regularization and Fine-tuning: To combat overfitting to synthetic data and improve generalization to real-world scenarios, dropout is applied to the inputs. Additionally, the model is fine-tuned on real IMU data (from the newly created DIP-IMU dataset) to further bridge the gap between synthetic and real data distributions.

Datasets

  • Training Data:
    • AMASS: A large combination of motion capture datasets used to synthesize IMU data.
  • DIP-IMU (New Dataset): A newly recorded dataset consisting of 10 subjects wearing 17 Xsens IMUs, capturing approximately 90 minutes of diverse activities. Ground truth SMPL poses were obtained by running the SIP method on the 17 IMU sensors. This dataset is made publicly available.
  • Evaluation Data:
    • TotalCapture: A benchmark dataset with synchronized IMU, motion capture, and RGB data.
  • DIP-IMU: Used for evaluating the model's performance on real-world data, especially for activities underrepresented in existing datasets.
  • Playground: Used for qualitative evaluation due to its challenging outdoor nature and uncommon poses.

Key Insights and Ideas

  • Synthetic Data is Key: Leveraging large motion capture datasets to synthesize IMU data is crucial for training robust deep learning models when real-world synchronized data is scarce.
  • Bi-directional RNNs for Temporal Context: Using BiRNNs significantly improves pose reconstruction by allowing the model to learn from both past and future motion dynamics, a critical factor for smooth and accurate predictions.
  • Auxiliary Acceleration Loss for Robustness: Explicitly training the network to reconstruct acceleration data helps it better handle noisy and ambiguous sensor readings, leading to improved pose accuracy, especially for challenging poses.
  • Real-time Performance with Sliding Window: A sliding window approach with a small look-ahead window effectively balances real-time performance with prediction accuracy, making the system practical for interactive applications.
  • Fine-tuning Bridges the Gap: Fine-tuning a model pre-trained on synthetic data with a smaller, real-world dataset significantly improves performance on unseen real-world scenarios, especially for activities not well-represented in the synthetic data.

Key Results

  • Accuracy: DIP achieves a mean joint angle error of 15.85° on the TotalCapture dataset, outperforming the offline baseline SIP. After fine-tuning on DIP-IMU, the error on this dataset drops to 17.54°, comparable to offline methods.
  • Real-time Performance: The system achieves real-time performance, running at approximately 29 fps on a single GPU, with a latency of only 85ms.
  • Qualitative Results: DIP produces smooth poses with fewer inter-penetrations compared to baselines, demonstrating that it learns a mapping to the space of valid human poses.
  • Dataset Contribution: The paper introduces DIP-IMU, the largest publicly available dataset of real IMU data for human pose reconstruction, along with the generated synthetic IMU data from AMASS.

Limitations and Future Work

  • Acceleration Modeling: The authors acknowledge that modeling accelerations (both synthetic and real) remains a challenge and is a primary reason for fine-tuning on real data. Future work could explore new sensor placement strategies to mitigate this.
  • Multi-person Interaction: The current model focuses on single-person pose estimation. Extending it to handle multi-person interactions and interactions with objects is a key area for future research.
  • Integration with Vision: Combining IMU data with visual input from body-mounted cameras could further improve pose estimation, particularly for contact points and poses like sitting.
  • Global Translation: DIP currently does not estimate global translation. Future work could incorporate GPS signals or learn global translation directly from IMU inputs.
  • Latency for AR: While real-time, the latency might still be a concern for highly interactive AR applications. Exploring methods to predict further into the future could reduce this.

Summary updated

Abstract

We demonstrate a novel deep neural network capable of reconstructing human full body pose in real-time from 6 Inertial Measurement Units (IMUs) worn on the user's body. In doing so, we address several difficult challenges. First, the problem is severely under-constrained as multiple pose parameters produce the same IMU orientations. Second, capturing IMU data in conjunction with ground-truth poses is expensive and difficult to do in many target application scenarios (e.g., outdoors). Third, modeling temporal dependencies through non-linear optimization has proven effective in prior work but makes real-time prediction infeasible. To address this important limitation, we learn the temporal pose priors using deep learning. To learn from sufficient data, we synthesize IMU data from motion capture datasets. A bi-directional RNN architecture leverages past and future information that is available at training time. At test time, we deploy the network in a sliding window fashion, retaining real time capabilities. To evaluate our method, we recorded DIP-IMU, a dataset consisting of 10 subjects wearing 17 IMUs for validation in 64 sequences with 330 000 time instants; this constitutes the largest IMU dataset publicly available. We quantitatively evaluate our approach on multiple datasets and show results from a real-time implementation. DIP-IMU and the code are available for research purposes.1
BibTeX
@article{huang2018deep,
  author = {Yinghao Huang and Manuel Kaufmann and Emre Aksan and Michael J. Black and Otmar Hilliges and Gerard Pons-Moll},
  title = {{Deep inertial poser: learning to reconstruct human pose from sparse inertial measurements in real time}},
  journal = {ACM Trans. Graphics},
  year = {2018},
  doi = {10.1145/3272127.3275108},
}

Loading related content...

Figure