Pedestrian Trajectory Prediction Using Dynamics-based Deep Learning

Honghui Wang, Weiming Zhi, Gustavo Batista, Rohitash Chandra

TL;DR

This paper introduces a "dynamics-based deep learning" (DDL) framework that integrates mathematically guaranteed stable dynamical systems into Transformer-based models for pedestrian trajectory prediction. By forcing predicted paths to converge toward a goal while learning social navigation behaviors, the model achieves a significant performance boost—outperforming state-of-the-art models by over 23% on standard benchmarks.

What Problem Does This Paper Address?

Predicting where a person will walk is a cornerstone of autonomous driving and social robotics. While deep learning models like RNNs and Transformers have become the standard, they suffer from two major flaws:

  1. The Black-Box Problem: They lack explainability. It’s often unclear why a model predicts a specific curved path.
  2. Lack of Physical Constraints: These models often ignore the fundamental truth that human movement is goal-directed. Without explicit constraints, predicted trajectories can be physically unrealistic or fail to converge on a logical destination.

The authors aim to bridge the gap between "physics-based" models (which are explainable but rigid) and "deep learning" models (which are flexible but opaque).

Methods

The researchers propose the Dynamics-based Deep Learning (DDL) framework, which consists of three main stages:

  • Goal Estimation: Using a $\gamma$-soft Dynamic Time Warping (DTW) algorithm and K-means clustering, the model looks at a pedestrian's history and compares it to an "expert repository" of past trajectories to estimate their likely destination (the "equilibrium point").
  • Asymptotically Stable Dynamical System (DS): Instead of predicting raw coordinates, the framework uses a DS defined by a stable attractor. Mathematically, this ensures that the trajectory will always converge to the predicted goal.
  • Transformer-based Learning (STAR): The authors use the Spatio-Temporal grAph tRansformer (STAR) to learn the parameters of a positive-definite matrix ($P$). This matrix "shapes" the dynamical system, allowing the model to learn complex behaviors like curving to avoid obstacles or maintaining social distance, while the underlying DS math guarantees stability.

To ensure the matrix $P$ remains positive-definite (a requirement for stability), they employ an LU decomposition strategy during training.

Datasets

The model was evaluated on the ETH and UCY datasets, which are the industry-standard benchmarks for pedestrian trajectory prediction. These datasets contain real-world video data of pedestrians in various settings (university campuses, sidewalks, etc.) converted into 2D coordinates.

Key Insights and Ideas

  • Stability as a Prior: By integrating Lyapunov stability theory directly into the neural network's architecture, the authors provide the model with "prior knowledge" that humans walk toward destinations. This reduces the search space for the model and prevents "wandering" predictions.
  • Explainability through Velocity: Because the model predicts velocity based on a gradient descent toward a goal, the resulting trajectories are inherently more interpretable. We can visualize the "force" pulling the pedestrian toward their destination.
  • Implicit Social Norms: Even though the DS is a mathematical construct, the Transformer learns to adjust the $P$ matrix based on the surrounding environment. This allows the model to mimic social behaviors—like slowing down or veering slightly to avoid a collision—without needing hand-coded "social force" rules.

Key Results

The DDL framework demonstrated superior performance across five benchmark datasets:

  • Accuracy Gains: Compared to the baseline STAR model, DDL showed an average improvement of 23.1% in Average Displacement Error (ADE) and 26.4% in Final Displacement Error (FDE).
  • SOTA Comparison: DDL outperformed prominent models such as PECNet, Goal-GAN, and Social-STGCNN. For example, on the ZARA2 dataset, it achieved an ADE of 0.13 compared to STAR's 0.22.
  • Qualitative Success: Visualization showed the model successfully predicting "collision avoidance" behaviors, such as a pedestrian slowing down or changing direction when their path was obstructed by others.

Limitations and Future Work

The authors acknowledge a geometric limitation: because they use positive-definite matrices to ensure stability, the model is currently limited in how sharply it can change a pedestrian's motion angle.

Future Directions: * Diffeomorphisms: The authors suggest using "diffeomorphic transforms" to allow for more complex, highly curved trajectories while maintaining stability. * Broader Social Contexts: Testing the model in more diverse and crowded social settings to see how well the learned "shaping matrix" handles extreme congestion.

Summary updated

Abstract

Pedestrian trajectory prediction plays an important role in autonomous driving systems and robotics. Recent work utilizing prominent deep learning models for pedestrian motion prediction makes limited a priori assumptions about human movements, resulting in a lack of explainability and explicit constraints enforced on predicted trajectories. We present a dynamics-based deep learning framework with a novel asymptotically stable dynamical system integrated into a Transformer-based model. We use an asymptotically stable dynamical system to model human goal-targeted motion by enforcing the human walking trajectory, which converges to a predicted goal position, and to provide the Transformer model with prior knowledge and explainability. Our framework features the Transformer model that works with a goal estimator and dynamical system to learn features from pedestrian motion history. The results show that our framework outperforms prominent models using five benchmark human motion datasets.
BibTeX
@inproceedings{wang2024pedestrian,
  author = {Honghui Wang and Weiming Zhi and Gustavo Batista and Rohitash Chandra},
  title = {{Pedestrian Trajectory Prediction Using Dynamics-based Deep Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  url = {https://openreview.net/forum?id=NCQqai9FdQ},
}
Figure