Trajectory Unified Transformer for Pedestrian Trajectory Prediction

Liushuai Shi, Le Wang, Sanping Zhou, Gang Hua

Abstract

Pedestrian trajectory prediction requires modeling both individual multimodality and complex social interactions. Traditional state-of-the-art models often rely on computationally expensive post-processing steps, such as K-means clustering, to generate final predictions. We present TUTR (Trajectory Unified TRansformer), an architecture that unifies these components into a single Transformer encoder-decoder framework. TUTR eliminates the need for non-parallel post-processing, achieving state-of-the-art accuracy while operating 10x to 40x faster than previous top-tier methods.

Introduction

Predicting pedestrian movement is a fundamental challenge for autonomous systems, primarily due to the inherent multimodality of human intent and the influence of social interactions. Existing models typically generate a large set of potential trajectories and then apply clustering techniques to select a representative subset. However, this approach introduces significant latency, particularly in crowded environments, and often fails to provide reliable probability estimates for each path. TUTR addresses these limitations by treating trajectory prediction as a set-prediction problem, similar to modern object detection frameworks.

Methodology: The TUTR Architecture

TUTR utilizes a Transformer-based architecture to process trajectories through four distinct stages, bypassing traditional sampling-and-clustering bottlenecks.

The TUTR architecture consists of Global Prediction, Mode-Level Encoder, Social-Level Decoder, and a Dual Prediction Head.
  1. Global Prediction: The model identifies common "motion modes" (e.g., straight, left turn) from training data. These modes serve as initial tokens for the Transformer.
  2. Mode-Level Encoder: This stage models the relationships between different motion modes to capture the variety of possible human movements.
  3. Social-Level Decoder: The decoder incorporates "neighbor embeddings" to account for nearby pedestrians, ensuring that predicted paths are socially acceptable and collision-free.
  4. Dual Prediction Head: TUTR outputs both diverse trajectories and their associated probabilities in a single parallel step. This design enables the model to optimize for Brier metrics, which reward accurate probability calibration.

Experimental Evaluation

TUTR was evaluated on standard benchmarks, including the ETH/UCY datasets and the Stanford Drone Dataset (SDD). These datasets provide a range of real-world scenarios, from simple bird's-eye views to complex campus environments with high pedestrian density.

Results and Discussion

TUTR demonstrates a superior balance between accuracy and computational efficiency.

Performance comparison (A) and inference time scaling (B) of TUTR versus baseline models.

Accuracy and Probability Calibration

TUTR achieved state-of-the-art results on Brier-ADE and Brier-FDE metrics. Ablation studies confirmed that the "Global Prediction" stage is critical; replacing pre-calculated motion modes with standard learnable vectors resulted in a significant degradation of performance (e.g., ADE increased from 7.76 to 17.26 on SDD).

Computational Efficiency

A key advantage of TUTR is its scalability. In dense scenes with over 80 pedestrians, TUTR is approximately 40x faster than models like MemoNet and SocialVAE. By shifting the diversity logic into the parallelizable Transformer architecture, TUTR removes the CPU-bound bottleneck of clustering algorithms.

Conclusion

TUTR represents a significant step forward in real-time trajectory prediction by unifying multimodal and social modeling into a single, efficient Transformer framework. While highly effective, future iterations could benefit from incorporating environmental map data to further refine predictions in spatially constrained areas.

Summary updated

Abstract

Pedestrian trajectory prediction is an essential link to understanding human behavior. Recent work achieves state-of-the-art performance gained from hand-designed post-processing, e.g., clustering. However, this post-processing suffers from expensive inference time and neglects the probability that the predicted trajectory disturbs downstream safety decisions. In this paper, we present Trajectory Unified TRansformer, called TUTR, which unifies the trajectory prediction components, social interaction, and multimodal trajectory prediction, into a transformer encoder-decoder architecture to effectively remove the need for post-processing. Specifically, TUTR parses the relationships across various motion modes using an explicit global prediction and an implicit mode-level transformer encoder. Then, TUTR attends to the social interactions with neighbors by a social-level transformer decoder. Finally, a dual prediction forecasts diverse trajectories and corresponding probabilities in parallel without post-processing. TUTR achieves state-of-the-art accuracy performance and improvements in inference speed of about 10× - 40× compared to previous well-tuned state-of-the-art methods using post-processing.
BibTeX
@inproceedings{shi2023trajectory,
  author = {Liushuai Shi and Le Wang and Sanping Zhou and Gang Hua},
  title = {{Trajectory Unified Transformer for Pedestrian Trajectory Prediction}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00887},
}
Figure