Trajectorylet-Net: a novel framework for pose prediction based on trajectorylet descriptors

Xiaoli Liu, Jianqin Yin, Jin Tang, Zhicheng ZhangarXiv 2019 · arXiv

Abstract

Predicting future human motion is a critical task for human-robot interaction and autonomous systems. Traditional approaches often rely on Recurrent Neural Networks (RNNs), which can struggle with complex spatial hierarchies and long-term temporal dependencies. This paper introduces TrajectoryNet (or Trajectorylet-Net), a 2D Convolutional Neural Network (CNN) framework that redefines motion prediction by operating in a "trajectory space." By mapping temporal sequences to feature channels, the model effectively captures coupled spatio-temporal dynamics and hierarchical joint correlations. Our approach achieves state-of-the-art performance on major benchmarks, including Human3.6M, demonstrating significant improvements in short-term prediction accuracy.

Introduction

Human motion is characterized by intricate spatial and temporal dependencies. Existing research identifies three primary challenges in modeling these dynamics:

  1. Coupled Dynamics: Spatial joint positions and temporal transitions are intrinsically linked; processing them separately can break the natural state of motion.
  2. Hierarchical Spatial Correlations: Local correlations exist within limbs (e.g., elbow to wrist), while global correlations span the entire skeleton (e.g., left hand to right foot).
  3. Global Temporal Co-occurrence: Complex actions involve simultaneous sub-motions that are difficult to capture with sequential frame-by-frame processing.

Trajectorylet-Net addresses these gaps by moving away from sequential RNNs in favor of a hierarchical 2D CNN architecture that treats the entire motion sequence as a unified spatio-temporal volume.

Methodology: The Trajectory Space

The core innovation of Trajectorylet-Net is the Trajectory Space Transformation. Instead of processing frames one by one, the input sequence of 3D joint coordinates is mapped into a trajectory space using a $1 \times 1$ convolutional layer. In this representation, the temporal dimension is reorganized into the depth (channel) dimension. This allows subsequent layers to observe the entire trajectory of a joint simultaneously, facilitating the extraction of global temporal co-occurrence features.

Architecture

The network follows an encoder-decoder structure designed to preserve spatial hierarchy:

  • Encoder: Comprises four "trajectory blocks" that utilize 2D convolutions to extract features across growing receptive fields. This enables the model to learn local joint relationships before capturing global skeletal dynamics. Symmetrical residual connections ensure that fine-grained local details are preserved.
  • Decoder: Reconstructs the future pose sequence using $3 \times 3$ and $1 \times 1$ convolutions to ensure spatial consistency and temporal smoothness.
The Trajectorylet-Net architecture featuring the trajectory space transformation, encoder with trajectory blocks, and decoder.

Experimental Results

The model was evaluated on the Human3.6M (H3.6M), G3D, and FNTU datasets. Trajectorylet-Net consistently outperformed previous state-of-the-art (SOTA) methods, particularly in short-term horizons (80ms to 400ms).

On the H3.6M dataset, the model achieved an average Mean Per Joint Position Error (MPJPE) of 10.2mm at 80ms, a notable improvement over the 12.1mm achieved by previous methods like LearnTrajDep. The model also demonstrated strong generalization capabilities in cross-dataset testing, suggesting it captures fundamental physical laws of human motion rather than dataset-specific patterns.

Quantitative comparison of TrajectoryNet against other methods across different actions and time horizons.

Discussion and Conclusion

By treating time as a channel dimension, Trajectorylet-Net leverages the strengths of 2D CNNs—parallelization and hierarchical feature extraction—to surpass the limitations of RNNs in spatial modeling. Modeling joint trajectories rather than static positions provides the network with richer physical context for learning underlying motion dynamics.

Despite its success, challenges remain. Like many deterministic models, error accumulation leads to "mean pose" drift over long durations (>1s), where the skeleton eventually settles into a neutral position. Future work may involve integrating probabilistic components to account for the multi-modal nature of human intent, where a single starting pose could lead to multiple valid future actions.

Summary updated

BibTeX
@misc{liu2019trajectorylet,
  author = {Xiaoli Liu and Jianqin Yin and Jin Tang and Zhicheng Zhang},
  title = {{Trajectorylet-Net: a novel framework for pose prediction based on trajectorylet descriptors}},
  year = {2019},
  eprint = {1910.06583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.06583},
}
Figure