Improving Full-Body Pose Estimation from a Small Sensor Set Using Artificial Neural Networks and a Kalman Filter

Frank J. Wouda, Matteo Giuberti, Giovanni Bellusci, Bert-Jan F. van Beijnum, Peter H. VeltinkAAAI 2019 · DOI

TL;DR

Estimating full-body motion from a sparse sensor set often results in jittery, unrealistic movements because neural networks typically process frames in isolation. This research introduces a hybrid pipeline that passes Artificial Neural Network (ANN) predictions through an error-state Kalman Filter (KF). By enforcing temporal consistency, the system successfully smooths motion while maintaining a minimal sensor footprint.

The Problem: Sparse Sensors and Temporal Jitter

High-fidelity motion capture traditionally requires extensive sensor arrays—either dozens of optical markers or 17+ Inertial Measurement Units (IMUs). This setup is cumbersome and intrusive. To simplify this, researchers use ANNs to estimate a full 23-segment body pose from as few as four sensors. However, these models often lack temporal coherence. Because the network evaluates each frame independently, the resulting avatar may "teleport" or jitter, creating physically impossible movements.

Methods: A Two-Stage Hybrid Pipeline

The proposed solution uses a two-stage architecture to bridge the gap between sparse data and realistic motion.

Pipeline architecture showing the flow from minimal sensor input through the ANN and Kalman Filter.
  1. The ANN Mapper: An ANN (two hidden layers of 200 and 100 neurons) maps the orientations of four segments (lower legs and arms) relative to the pelvis into a full-body pose.
  2. The Error-State Kalman Filter (KF): The ANN’s output is treated as a "noisy measurement" and fed into a KF. Standard KFs struggle with non-linear quaternions; thus, the authors employed an error-state KF. This tracks the difference (error) between orientations, keeping the mathematics linear and stable. A "constant velocity" model acts as a prior to prevent sudden, unrealistic jumps.

Datasets

The system was validated using a dataset from 6 participants, totaling approximately 120 minutes of movement. Ground truth was established using a professional Xsens MVN system with 17 sensors. The ANN was specifically trained to reconstruct this full 17-sensor data from a 4-sensor subset.

Key Insights

  • Physics as a Filter: The KF acts as a bridge between the "black box" ANN and physical reality, enforcing temporal constraints without requiring the neural network itself to understand physics.
  • Robust Rotation Handling: By using an error-state KF and helical angles, the authors found a mathematically stable way to smooth quaternions, avoiding common linearization errors.
  • Visual Realism vs. Raw Accuracy: A system can be numerically accurate on average but visually unusable due to jitter. Smoothing is often more critical for realism than minimizing absolute mean error.

Results: Smoothing the Spikes

The most significant outcome was the visual stabilization of the pose. As shown in the error analysis, the "spiky" errors characteristic of raw ANN output—particularly in the knees where local sensors were absent—were significantly mitigated.

Comparison of knee joint position error over time between the raw ANN estimate (ML) and the Kalman Filter output (OE).

Quantitatively, the mean joint position error decreased from 0.122m (ANN only) to 0.111m (ANN + KF). While the numerical gain is modest, the reduction in high-frequency jitter represents a substantial improvement in motion quality.

Limitations and Future Work

The current model uses a static noise covariance matrix. However, jitter often correlates with specific poses, such as the end of a swing phase in walking. Implementing a time-variant noise matrix could allow the system to "distrust" the ANN more during high-uncertainty phases. Additionally, while the constant velocity assumption works for small timesteps, it may struggle with high-acceleration athletic movements. Future iterations could also incorporate explicit biomechanical constraints, such as joint limits, to further ensure physical plausibility.

Summary updated

BibTeX
@inproceedings{wouda2019improving,
  author = {Frank J. Wouda and Matteo Giuberti and Giovanni Bellusci and Bert-Jan F. van Beijnum and Peter H. Veltink},
  title = {{Improving Full-Body Pose Estimation from a Small Sensor Set Using Artificial Neural Networks and a Kalman Filter}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.1609/aaai.v33i01.330110063},
}
Figure