Dynamical Deep Generative Latent Modeling of 3D Skeletal Motion

Amirreza Farnoosh, Sarah Ostadabbas

Introduction

Analyzing 3D motion capture (MoCap) data is a cornerstone of biomechanics, sports science, and robotics. However, this data is notoriously difficult to handle: it is high-dimensional, highly correlated across joints, and non-linear. Crucially, biological motion is "switching"—an animal doesn't just move randomly; it transitions between distinct regimes, such as a bird switching from flapping to gliding or a human transitioning between gait phases.

Existing tools generally fall into two camps. Clustering methods can group similar poses but lack an understanding of the underlying physics or dynamics, limiting their predictive power. Conversely, linear dynamical systems understand movement but assume transitions are simple and linear, failing to capture the complex, high-velocity dynamics of a golf swing or the aerodynamics of animal flight. This paper bridges this gap by introducing a model that is both generative and interpretable.

Methods: Bayesian Switching Dynamical Models

The core of the proposed approach is a Bayesian switching dynamical model that decomposes complex 3D skeletal motion into interpretable, low-dimensional "states." The model architecture is built on three key pillars:

  1. Low-Dimensional Latent Space: Instead of modeling every 3D coordinate of every joint directly, the model projects the high-dimensional data into a small "latent" space that captures the essence of the movement.
  2. Switching Deep VAR Prior: This is the central innovation. The model assumes motion is governed by a set of discrete states ($S$). Within each state, the next movement is predicted by a Deep Vector Autoregressive (VAR) process. Unlike standard models that only look at the previous frame, this architecture utilizes multi-layer perceptrons (MLPs) to handle non-linearities and can incorporate higher-order lags (looking back multiple frames).
  3. Variational Inference: To train the model, the authors employ approximate variational inference by maximizing the Evidence Lower Bound (ELBO), allowing for computationally efficient learning of complex patterns.
Graphical model of the proposed Bayesian switching dynamical system, showing the relationships between discrete states (s), latent variables (z), and observed data (x).

Key Insights

By modeling the change in motion rather than just static poses, the model discovers biologically valid states. For example, in walking data, it automatically identifies "right-leg swing" versus "left-leg swing" without prior anatomical knowledge.

A crucial finding from the authors' ablation studies is the importance of "memory." Higher-order lags (looking back 2+ frames) are critical for physical systems. For a pendulum, predicting the future requires knowledge of both position and velocity; a first-order model that only considers the current frame will inherently struggle with such physics-based motion.

Experimental Results

The model was evaluated on a diverse range of motions, including a simulated pendulum, human salsa dancing, walking, golf swings, and bat flight. Performance was measured using Normalized Root-Mean-Square Error (NRMSE%) for temporal prediction.

  • Superior Accuracy: The proposed model significantly outperformed state-of-the-art baselines. For the pendulum system, it achieved a 5.95% error rate, compared to 25.58% for the linear rSLDS model.
  • Interpretability and Segmentation: The model successfully segmented complex trials into meaningful phases. In the Salsa dataset, it identified states corresponding to clockwise turns, anticlockwise turns, and twirling.
  • Robustness: The model successfully reconstructed bat flight sequences even when over 32% of joint markers were missing due to occlusions.
Segmentation of human motion into discrete, interpretable states. The color-coded bar represents different dynamical regimes discovered by the model, mapped to specific skeletal pose sequences.

Conclusion and Future Work

This deep generative latent model provides a powerful framework for analyzing and generating complex 3D motion. While highly effective, future work could address the current need to pre-define the number of discrete states and the latent dimensionality. Automating this discovery process and adapting the model for real-time inference in rehabilitation or sports training are promising next steps.

Summary updated

BibTeX
@article{farnoosh2022dynamical,
  author = {Amirreza Farnoosh and Sarah Ostadabbas},
  title = {{Dynamical Deep Generative Latent Modeling of 3D Skeletal Motion}},
  journal = {Int. J. Comput. Vis.},
  year = {2022},
  doi = {10.1007/s11263-022-01668-8},
}
Figure