Motion reconstruction using sparse accelerometer data

Jochen Tautges, Arno Zinke, Björn Krüger, Jan Baumann, Andreas Weber, Thomas Helten, Meinard Müller, Hans-Peter Seidel, Bernd EberhardtACM Trans. Graphics 2011 · DOI

TL;DR

This paper introduces a novel framework for reconstructing full-body human motion from a sparse set of just four 3D accelerometers. By leveraging a large database of motion capture data and an efficient online retrieval system, the method can generate visually plausible animations even with limited sensor input and moderate variations in motion speed.

What Problem Does This Paper Address?

Generating realistic human animations is crucial for applications like virtual reality, gaming, and film. Traditional motion capture (mocap) systems, while accurate, are often expensive, cumbersome, and require controlled environments. This paper tackles the challenge of achieving high-quality motion reconstruction using a much simpler, more accessible, and less restrictive approach: sparse accelerometer data. The core problem is how to infer complex, high-dimensional full-body motion from a few low-dimensional sensor readings that are inherently noisy and provide only partial information about the overall pose and movement. Existing methods often struggle with the ambiguity of sparse sensor data, temporal variations, and the need for large, diverse motion databases.

Methods

The authors propose a data-driven approach that combines an efficient online retrieval system with a novel prior model for motion synthesis.

  • Sparse Sensor Input: The system uses four 3D accelerometers placed on the wrists and ankles. This is a significant reduction compared to many existing systems.
  • Knowledge Base: A large database of pre-recorded motion capture data serves as the "knowledge base." This database is pre-processed to simulate accelerometer readings from each motion clip, allowing for cross-modal retrieval.
  • Online Lazy Neighborhood Graph (OLNG): This is a key innovation. It's an incremental data structure that efficiently tracks the K nearest neighbors of recent accelerometer readings. Unlike static graphs, the OLNG updates dynamically, allowing for real-time processing and handling of temporal variations without being constrained by a fixed window size. This significantly speeds up retrieval and reduces memory usage.
  • Motion Reconstruction: At runtime, the system uses the OLNG to retrieve relevant motion fragments from the knowledge base. These fragments are then used to synthesize the final animation.
  • Prior Model: To address the inherent ambiguity of sparse accelerometer data, the authors introduce a novel, data-driven prior model. This model consists of three components:
  • Pose Prior: Uses kernel regression to estimate the likelihood of a pose based on the distribution of poses in the retrieved motion fragments.
  • Motion Prior: Accounts for the temporal evolution of motion by considering joint velocities and accelerations from the retrieved fragments.
  • Smoothness Prior: Enforces smooth motion by ensuring that the synthesized joint accelerations are consistent with those in the database.
  • Energy Minimization: The final pose is determined by minimizing an energy function that balances the control term (driven by accelerometer data) and the prior term (ensuring plausibility and consistency with the database). A gradient descent method is used for optimization, often with a dimensionality reduction step using PCA for efficiency.

Datasets

  • HDM05 Database: This publicly available motion capture database was used extensively for creating the knowledge base and for evaluation. The authors specifically mention using various parts and sections of this database, performed by different actors.
  • Simulated Sensor Data: The authors generated simulated accelerometer data from the HDM05 motion capture data to serve as control input for their system during testing. This allowed for controlled experiments and comparisons with ground truth.
  • Real Sensor Data: The system was also tested with real accelerometer readings from an MTx device attached to a user's wrist, demonstrating its applicability in real-world scenarios.

Key Insights and Ideas

  • Sparse is Sufficient: The paper convincingly demonstrates that a surprisingly small number of accelerometers (four) can be sufficient for reconstructing complex full-body motion, significantly lowering the barrier to entry for motion capture.
  • OLNG for Efficiency and Adaptability: The OLNG is a crucial contribution, enabling real-time performance and adaptability to temporal variations that traditional methods struggle with. Its incremental nature is key to its online capability.
  • Data-Driven Priors are Powerful: The novel prior model, particularly its kernel regression-based approach, is highly effective in resolving ambiguities and ensuring plausible motion synthesis, even when retrieved fragments come from diverse motion types. This moves beyond simple smoothness constraints.
  • Database Diversity Matters: The quality of reconstruction is significantly influenced by the diversity and size of the motion database, especially when the actor being reconstructed is not present in the database.

Key Results

  • High-Quality Reconstruction: The system successfully reconstructs full-body motion from sparse accelerometer data, producing visually plausible animations.
  • Efficiency Gains: The OLNG provides a speedup of over an order of magnitude compared to previous static neighborhood graph approaches.
  • Robustness to Temporal Variations: The OLNG allows for synthesizing motions at speeds not explicitly present in the knowledge base, demonstrating robustness to moderate time warping.
  • Accuracy Improvements: The novel prior model leads to a significant reduction in average reconstruction error (around 30%) compared to adapting existing prior models.
  • Sensor Placement Impact: While more sensors generally improve accuracy, the results show that even with a few sensors (e.g., four), good results can be achieved, especially for motions with consistent styles across individuals.
  • Real vs. Simulated Data: The study found that real sensor readings and simulated accelerations yield comparable reconstruction results, validating the use of large simulated datasets for evaluation.

Limitations and Future Work

  • Root Motion Approximation: The current method only approximates root motion due to the lack of direct positional and orientational sensor data. More sophisticated sensors would be needed for precise world-frame movement.
  • Skeleton Consistency: The usability of the method is affected by the lack of consistent skeleton representations in existing mocap databases, which can impact the accuracy of joint frame calculations.
  • Real-time Performance: While efficient, the current MATLAB implementation is not yet real-time for all components (optimization is the bottleneck). Future work will focus on code optimization and porting to C++ for real-time applications.
  • Sensor Expansion: The authors plan to build devices with their chosen sensor setup for practical motion puppetry applications.
  • Broader Applications of Priors: The data-driven prior model has potential applications beyond motion reconstruction, such as synthesizing missing degrees of freedom in keyframe animations or "texturing" animations with motion details.

Summary updated

BibTeX
@article{tautges2011motion,
  author = {Jochen Tautges and Arno Zinke and Björn Krüger and Jan Baumann and Andreas Weber and Thomas Helten and Meinard Müller and Hans-Peter Seidel and Bernd Eberhardt},
  title = {{Motion reconstruction using sparse accelerometer data}},
  journal = {ACM Trans. Graphics},
  year = {2011},
  doi = {10.1145/1966394.1966397},
}
Figure