Convolutional Sequence to Sequence Model for Human Dynamics

Chen Li, Zhen Zhang, Wee Sun Lee, Gim Hee LeeOpenReview 2022 · OpenReview

TL;DR

This paper introduces a novel convolutional sequence-to-sequence model for human motion prediction that effectively captures both long-term and short-term temporal dynamics, as well as spatial correlations between body parts. By leveraging CNNs, their model significantly outperforms existing RNN-based methods, particularly in long-term predictions, by avoiding the common "mean pose" problem and generating more realistic human movements.

What Problem Does This Paper Address?

Predicting human motion is a crucial task for applications ranging from robotics and autonomous driving to sports analysis and medical diagnosis. However, human motion is inherently complex, characterized by high dimensionality and intricate, non-linear dynamics. Existing data-driven methods, particularly those based on Recurrent Neural Networks (RNNs), struggle with accurately capturing long-term dependencies and spatial relationships between different body parts. This often leads to predictions that become unrealistic over time, converging to a static "mean pose" or exhibiting jerky, unnatural movements, especially for complex or aperiodic actions. The challenge lies in developing a model that can simultaneously understand the overall action context (long-term) and the immediate, localized movements (short-term), while also respecting the biomechanical constraints of the human body.

Methods

The authors propose a Convolutional Sequence-to-Sequence (ConvSeq2Seq) model for human motion prediction. Unlike traditional RNN-based sequence-to-sequence models, their approach utilizes Convolutional Neural Networks (CNNs) to process the motion sequences. The core of their architecture is a Convolutional Encoding Module (CEM), which is designed to hierarchically capture spatial and temporal correlations.

Key components of their method include:

  • Long-Term Encoder: A CEM that encodes the entire input motion sequence into a single "long-term hidden variable" ($z_l$). This variable is intended to capture invariant information about the action category, body properties, and environmental context.
  • Short-Term Encoder: Another CEM that operates on a sliding window of recent frames (e.g., the last $C$ frames) to produce a "short-term hidden variable" ($z_s$). This captures immediate, dynamic information.
  • Spatial Decoder: This component takes the concatenated long-term and short-term hidden variables and predicts the next pose. It also has a residual connection, meaning it predicts the change in pose rather than the absolute pose, which has been shown to improve prediction accuracy.
  • Recursive Prediction: The short-term encoder and decoder are applied recursively. After predicting a frame, the window slides forward, and the process repeats to generate the entire future sequence.
  • Rectangular Convolutional Kernels: To better capture spatial correlations between different body parts, they employ rectangular kernels (e.g., 2x7) in their CEM, allowing for a larger receptive field in the spatial domain.
  • Adversarial Regularization: To improve the qualitative realism of the generated motions, they incorporate an adversarial regularizer, inspired by Generative Adversarial Networks (GANs).

Datasets

The proposed model was trained and evaluated on two prominent human motion datasets:

  • Human3.6M dataset: This is described as the largest available video pose dataset, providing accurate 3D body joint locations. It features a wide variety of poses and actions, making it a challenging benchmark.
  • CMU Motion Capture dataset: This dataset was used to test the model's generalization ability across different categories of human actions.

Key Insights and Ideas

  • CNNs for Spatio-Temporal Dynamics: The paper demonstrates that the hierarchical structure of CNNs is naturally suited for capturing both the spatial relationships between body joints and the temporal dependencies in human motion, outperforming RNNs which primarily excel at sequential data.
  • Decoupling Long-Term and Short-Term Information: By using separate encoders for long-term context and short-term dynamics, the model can effectively leverage both global action understanding and immediate movement cues, leading to more robust predictions.
  • Avoiding the "Mean Pose" Problem: The authors argue that RNNs' difficulty in maintaining long-term information leads to the "mean pose" problem. Their ConvSeq2Seq architecture, by explicitly encoding long-term context, mitigates this issue, producing more dynamic and realistic future poses.
  • Spatial Kernel Design Matters: The use of rectangular convolutional kernels (2x7) proved more effective than square kernels for capturing the complex spatial correlations between different limbs in human motion.

Key Results

  • Quantitative Performance: The proposed model consistently outperforms state-of-the-art RNN-based methods (ERD, LSTM-3LR, SRNN, RRNN, LSTM-AE) on both the Human3.6M and CMU Motion Capture datasets across various prediction horizons (80ms to 1000ms). For instance, on Human3.6M, their model achieves lower Euler angle errors compared to RRNN, especially for longer prediction times.
  • Qualitative Improvements: The model generates more realistic and natural human movements, particularly for aperiodic actions like "eating," "smoking," and "discussion," where RNN-based methods tend to converge to unrealistic mean poses. The paper includes qualitative examples (Figure 3) showing their model's ability to maintain gestures and avoid unnatural poses.
  • Ablation Studies: Removing the long-term encoder significantly increases prediction error, highlighting its importance. The adversarial regularizer also shows marginal but noticeable improvements in qualitative performance.

Limitations and Future Work

  • Window Size Trade-off: The choice of the short-term window size ($C$) involves a trade-off between capturing sufficient short-term information and computational cost. While they explored different sizes, the optimal value might depend on the specific application.
  • Computational Resources: While the paper mentions their model is implemented in TensorFlow and uses ADAM optimizer, specific details on training time or computational requirements are not extensively detailed, though they do mention the recursive nature of the decoder.
  • Future Work: The paper implicitly suggests that further exploration of kernel designs and architectural variations within the ConvSeq2Seq framework could lead to even better performance. The success in avoiding the mean pose problem for complex actions also opens doors for more challenging motion prediction tasks.

Summary updated

Abstract

Human motion modeling is a classic problem in com- puter vision and graphics. Challenges in modeling human motion include high dimensional prediction as well as ex- tremely complicated dynamics.We present a novel approach to human motion modeling based on convolutional neural networks (CNN). The hierarchical structure of CNN makes it capable of capturing both spatial and temporal correla- tions effectively. In our proposed approach, a convolutional long-term encoder is used to encode the whole given motion sequence into a long-term hidden variable, which is used with a decoder to predict the remainder of the sequence. The decoder itself also has an encoder-decoder structure, in which the short-term encoder encodes a shorter sequence to a short-term hidden variable, and the spatial decoder maps the long and short-term hidden variable to motion predic- tions. By using such a model, we are able to capture both invariant and dynamic information of human motion, which results in more accurate predictions. Experiments show that our algorithm outperforms the state-of-the-art methods on the Human3.6M and CMU Motion Capture datasets. Our code is available at the project website
BibTeX
@misc{li2022convolutional,
  author = {Chen Li and Zhen Zhang and Wee Sun Lee and Gim Hee Lee},
  title = {{Convolutional Sequence to Sequence Model for Human Dynamics}},
  year = {2022},
  url = {https://openreview.net/forum?id=djwUhRnYSaE},
  howpublished = {OpenReview},
}
Figure