Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction

Kun Qian, Abduallah Mohamed, Mohamed Elhoseiny, Christian Claudel

Abstract

Social-STGCNN is a novel deep learning architecture that treats pedestrian trajectory prediction as a graph problem rather than a sequence-to-sequence problem. By replacing heavy recurrent units (LSTMs) and heuristic pooling layers with a Spatio-Temporal Graph Convolutional Neural Network (ST-GCNN) and a "Time-Extrapolator" CNN (TXP-CNN), the model achieves state-of-the-art accuracy with 8.5x fewer parameters and up to 48x faster inference speeds.

Introduction

Predicting human movement in crowded environments is a significant challenge for autonomous systems. Human paths are governed by "social conventions": we walk in groups, avoid collisions, and adjust our speed based on those around us. Traditional state-of-the-art models, such as Social-LSTM and Social-GAN, attempt to solve this by modeling each individual with a Recurrent Neural Network (RNN) and aggregating social context through heuristic pooling layers.

However, these methods suffer from three major flaws: computational inefficiency due to the sequential nature of RNNs, information loss in pooling layers, and high model complexity. Social-STGCNN addresses these by modeling the scene as a graph where social interactions are explicitly represented through edges.

Overview of the Social-STGCNN model pipeline, showing the transition from observed trajectories to a single-pass prediction.

Methodology

The Social-STGCNN architecture moves away from the "one RNN per person" paradigm, utilizing a two-component system designed for speed and structural integrity.

1. Spatio-Temporal Graph Convolutional Neural Network (ST-GCNN)

Instead of a list of coordinates, the scene is modeled as a graph where each pedestrian is a node and edges represent social influence. The authors introduce a novel weighted adjacency matrix using a kernel function based on the inverse of the distance between pedestrians. This allows the model to mathematically represent the degree of influence one person has on another's path. The ST-GCNN extracts features that capture both spatial (social) and temporal (movement) dynamics simultaneously.

2. Time-Extrapolator CNN (TXP-CNN)

A radical departure from previous work, the TXP-CNN replaces the step-by-step prediction of RNNs with a series of Temporal Convolutional Neural Networks (TCNs). This component operates on the temporal dimension of the graph embedding in a single pass, making it significantly faster and more parameter-efficient than recurrent architectures.

Detailed architecture of the Social-STGCNN, illustrating the graph representation, ST-GCNN layers, and the TXP-CNN for final distribution prediction.

Experiments and Results

The model was evaluated on the standard ETH and UCY benchmark datasets, which contain real-world pedestrian coordinates in various crowded environments.

Key Insights

  • Efficiency of Graph Representation: By representing the scene as a graph, the social structure is baked into the architecture, eliminating the need for complex pooling layers.
  • The "Single Pass" Advantage: Using CNNs for both spatial and temporal features allows the model to predict the entire future trajectory at once, preventing the error accumulation common in RNNs.
  • Parameter Efficiency: The model uses only 7.6K parameters, compared to 46.3K for Social-GAN and 264K for Social-LSTM, demonstrating that capturing the right structure is more important than raw capacity.

Quantitative Performance

  • Accuracy: Improved the Final Displacement Error (FDE) by 20% over previous state-of-the-art models.
  • Speed: Inference time is approximately 0.002 seconds per step, which is 48x faster than Social-GAN.
  • Data Efficiency: The model can outperform previous benchmarks on the Average Displacement Error (ADE) metric using only 20% of the training data.

Conclusion

Social-STGCNN demonstrates that graphs are the natural language of social interaction for trajectory prediction. By moving to a fully convolutional, graph-based architecture, the model achieves superior performance with a fraction of the computational cost. Future work may involve extending the graph to include multi-modal entities like bicycles and cars, as well as incorporating environmental context such as walls and stationary obstacles.

Summary updated

Abstract

Better machine understanding of pedestrian behaviors enables faster progress in modeling interactions between agents such as autonomous vehicles and humans. Pedestrian trajectories are not only influenced by the pedestrian itself but also by interaction with surrounding objects. Previous methods modeled these interactions by using a variety of aggregation methods that integrate different learned pedestrians states. We propose the Social Spatio-Temporal Graph Convolutional Neural Network (Social-STGCNN), which substitutes the need of aggregation methods by modeling the interactions as a graph. Our results show an improvement over the state of art by 20% on the Final Displacement Error (FDE) and an improvement on the Average Displacement Error (ADE) with 8.5 times less parameters and up to 48 times faster inference speed than previously reported methods. In addition, our model is data efficient, and exceeds previous state of the art on the ADE metric with only 20% of the training data. We propose a kernel function to embed the social interactions between pedestrians within the adjacency matrix. Through qualitative analysis, we show that our model inherited social behaviors that can be expected between pedestrians trajectories. Code is available at https://github.com/abduallahmohamed/Social-STGCNN.
BibTeX
@inproceedings{qian2020social,
  author = {Kun Qian and Abduallah Mohamed and Mohamed Elhoseiny and Christian Claudel},
  title = {{Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2020},
  url = {https://openreview.net/forum?id=Tuta7Gui6W5o},
}
Figure