Flow-based Autoregressive Structured Prediction of Human Motion

Mohsen Zand, Ali Etemad, Michael A. Greenspan

Abstract

Predicting complex spatio-temporal dynamics, such as human motion and multi-agent trajectories, requires models that can capture both deterministic rules and stochastic variability. We present MotionFlow, a novel generative framework based on Conditional Normalizing Flows (CNFs). MotionFlow introduces Locally Masked Convolutions (LMConv) for efficient, parallelized autoregressive conditioning and a dynamic latent space to model evolving temporal dependencies. Our approach effectively mitigates the "over-smoothing" problem common in traditional recurrent and adversarial models. MotionFlow achieves state-of-the-art results across ten diverse datasets, spanning human motion forecasting, trajectory prediction, and time-series analysis.

Introduction

Modeling the evolution of complex, interacting systems—ranging from the coordinated movement of human joints to the strategic navigation of basketball players—is a fundamental challenge in structured prediction. These systems are characterized by high-dimensional state spaces and inherent stochasticity.

Traditional deep learning approaches, typically combining Recurrent Neural Networks (RNNs) with Variational Autoencoders (VAEs) or Generative Adversarial Networks (GANs), often struggle with these complexities. Common failure modes include training instabilities like mode collapse and "over-smoothing," where models predict an average, unrealistic trajectory to minimize uncertainty. MotionFlow aims to overcome these limitations by providing a robust generative framework that maintains structural detail and realistic diversity.

Methodology

MotionFlow extends the capabilities of Conditional Normalizing Flows (CNFs) to the domain of spatio-temporal structured prediction. Normalizing flows are particularly well-suited for this task as they transform simple distributions into complex ones via invertible mappings, allowing for exact likelihood estimation and avoiding the pitfalls of GAN training.

Spatio-Temporal Conditioner

A core innovation in MotionFlow is the use of Locally Masked Convolutions (LMConv) instead of sequential RNNs for feature extraction. LMConv applies specific masks to enforce arbitrary spatial and temporal orderings within the input sequence. This convolutional approach allows the model to process historical frames in parallel during training, significantly improving computational efficiency while preserving autoregressive dependencies.

Comparison of masking and scanning patterns for PixelCNN, MaCow, and the proposed LMConv.

Dynamic Latent Space

To capture the evolving nature of motion, MotionFlow factorizes the latent space into frame-wise variables rather than a single static variable. A deep residual network models the temporal dependencies between these variables autoregressively. This dynamic representation allows the model to continuously update its internal state, which is crucial for preventing over-smoothing and ensuring diverse, realistic predictions.

Experiments and Results

The versatility of MotionFlow was evaluated across ten datasets covering four distinct tasks:

  • Trajectory Prediction: On the NBA dataset (multi-agent basketball tracking), MotionFlow achieved an Average Displacement Error (ADE) of 0.763m, outperforming strong graph-based baselines like EvolveGraph.
  • Human Motion Prediction: Using the CMU Mocap dataset, the model achieved state-of-the-art Mean Per Joint Position Error (MPJPE) across various actions. For instance, in the "basketball" action at a 1000ms horizon, it recorded an error of 84.27mm, surpassing the MSR-GCN baseline.
  • Time Series Forecasting: MotionFlow achieved top-tier Mean Squared Error (MSE) on six standard benchmarks, including Traffic and Electricity datasets.
  • Binary Segmentation: On the Weizmann Horse dataset, the model demonstrated its generalizability to purely spatial tasks, achieving 89.2% Mean IoU.

Discussion and Limitations

While MotionFlow demonstrates high performance, it requires a careful balance between deterministic constraints and stochastic hidden representations to ensure long-term prediction accuracy. Additionally, the model is sensitive to initialization; the authors noted that residual layers must be initialized to zero or trained with a very low learning rate to prevent gradient explosions.

Conclusion

MotionFlow represents a significant step forward in structured prediction by combining the strengths of normalizing flows with efficient spatio-temporal conditioning. By replacing sequential bottlenecks with parallelized masked convolutions and employing a dynamic latent space, the model generates highly realistic and diverse trajectories. Future work will focus on balancing deterministic and stochastic features for even higher-dimensional tasks such as video prediction.

Summary updated

BibTeX
@article{zand2023flow,
  author = {Mohsen Zand and Ali Etemad and Michael A. Greenspan},
  title = {{Flow-based Autoregressive Structured Prediction of Human Motion}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  doi = {10.1109/tpami.2023.3296446},
}
Figure