Abstract
Predicting future human motion is a critical task for human-robot interaction and autonomous systems. Traditional approaches often rely on Recurrent Neural Networks (RNNs), which can struggle with complex spatial hierarchies and long-term temporal dependencies. This paper introduces TrajectoryNet (or Trajectorylet-Net), a 2D Convolutional Neural Network (CNN) framework that redefines motion prediction by operating in a "trajectory space." By mapping temporal sequences to feature channels, the model effectively captures coupled spatio-temporal dynamics and hierarchical joint correlations. Our approach achieves state-of-the-art performance on major benchmarks, including Human3.6M, demonstrating significant improvements in short-term prediction accuracy.
Introduction
Human motion is characterized by intricate spatial and temporal dependencies. Existing research identifies three primary challenges in modeling these dynamics:
- Coupled Dynamics: Spatial joint positions and temporal transitions are intrinsically linked; processing them separately can break the natural state of motion.
- Hierarchical Spatial Correlations: Local correlations exist within limbs (e.g., elbow to wrist), while global correlations span the entire skeleton (e.g., left hand to right foot).
- Global Temporal Co-occurrence: Complex actions involve simultaneous sub-motions that are difficult to capture with sequential frame-by-frame processing.
Trajectorylet-Net addresses these gaps by moving away from sequential RNNs in favor of a hierarchical 2D CNN architecture that treats the entire motion sequence as a unified spatio-temporal volume.
Methodology: The Trajectory Space
The core innovation of Trajectorylet-Net is the Trajectory Space Transformation. Instead of processing frames one by one, the input sequence of 3D joint coordinates is mapped into a trajectory space using a $1 \times 1$ convolutional layer. In this representation, the temporal dimension is reorganized into the depth (channel) dimension. This allows subsequent layers to observe the entire trajectory of a joint simultaneously, facilitating the extraction of global temporal co-occurrence features.
Architecture
The network follows an encoder-decoder structure designed to preserve spatial hierarchy:
- Encoder: Comprises four "trajectory blocks" that utilize 2D convolutions to extract features across growing receptive fields. This enables the model to learn local joint relationships before capturing global skeletal dynamics. Symmetrical residual connections ensure that fine-grained local details are preserved.
- Decoder: Reconstructs the future pose sequence using $3 \times 3$ and $1 \times 1$ convolutions to ensure spatial consistency and temporal smoothness.
Experimental Results
The model was evaluated on the Human3.6M (H3.6M), G3D, and FNTU datasets. Trajectorylet-Net consistently outperformed previous state-of-the-art (SOTA) methods, particularly in short-term horizons (80ms to 400ms).
On the H3.6M dataset, the model achieved an average Mean Per Joint Position Error (MPJPE) of 10.2mm at 80ms, a notable improvement over the 12.1mm achieved by previous methods like LearnTrajDep. The model also demonstrated strong generalization capabilities in cross-dataset testing, suggesting it captures fundamental physical laws of human motion rather than dataset-specific patterns.
Discussion and Conclusion
By treating time as a channel dimension, Trajectorylet-Net leverages the strengths of 2D CNNs—parallelization and hierarchical feature extraction—to surpass the limitations of RNNs in spatial modeling. Modeling joint trajectories rather than static positions provides the network with richer physical context for learning underlying motion dynamics.
Despite its success, challenges remain. Like many deterministic models, error accumulation leads to "mean pose" drift over long durations (>1s), where the skeleton eventually settles into a neutral position. Future work may involve integrating probabilistic components to account for the multi-modal nature of human intent, where a single starting pose could lead to multiple valid future actions.
Summary updated