TrTr: A Versatile Pre-Trained Large Traffic Model based on Transformer for Capturing Trajectory Diversity in Vehicle Population

Ruyi Feng, Zhibin Li, Bowen Liu, Yan Ding, Ou ZhengarXiv 2023 · OpenReview

TL;DR

This paper introduces TrTr, a Transformer-based pre-trained model designed to capture the diverse behaviors of vehicle trajectories. By adapting Transformer's attention mechanism and introducing novel pre-training tasks with noise injection, TrTr effectively learns spatial and temporal patterns in traffic data, paving the way for improved trajectory prediction and other downstream traffic analysis tasks.

What Problem Does This Paper Address?

Understanding and modeling the diversity of vehicle trajectories is crucial for many traffic-related applications, such as predicting congestion, optimizing traffic flow, and ensuring safety. Traditional methods, including machine learning and recurrent neural networks (RNNs), struggle with this due to their need for large datasets and their sequential processing limitations, which can hinder the capture of complex, long-range dependencies in traffic data. The sheer number of parameters required for accurate modeling also presents a significant challenge.

Methods

The core of this work is the application of the Transformer architecture, known for its parallel processing capabilities and self-attention mechanism, to traffic trajectory data. The authors propose a novel approach that involves:

  • Data Structure Adaptation: Creating a specialized data structure that aligns with the Transformer's attention mechanism, allowing it to effectively process spatio-temporal information from vehicle trajectories.
  • Pre-training Tasks with Noise Injection: Designing specific pre-training tasks inspired by natural language processing (NLP) techniques. This includes introducing noise (e.g., masking vehicle positions) into the input data to encourage the model to learn robust spatial distributions and temporal dependencies. This noise injection is tailored to traffic scenarios, aiming to improve the model's ability to handle missing information and understand vehicle interactions.
  • Multi-Head Attention: Leveraging the Transformer's multi-head attention mechanism to capture intricate inter-vehicle relationships and spatio-temporal correlations within the traffic data.
  • Encoder-Decoder Architecture: Utilizing a standard Transformer encoder-decoder structure where the encoder acts as a feature extractor and the decoder generates the predicted trajectories.

Datasets

The paper utilizes the UTE dataset for training and evaluation. The data is partitioned into areas of 300 meters by 20 meters, with a maximum of 10 vehicles per area. The training data consists of approximately 400,000 steps per epoch, and the model is pre-trained for 50 epochs. The test dataset comprises 10,000 samples.

Key Insights and Ideas

  • Transformer's Suitability for Traffic: The paper demonstrates that the Transformer architecture, successful in NLP, can be effectively adapted to capture the complex spatio-temporal dynamics of vehicle trajectories.
  • Importance of Pre-training Tasks: The design of specific pre-training tasks, particularly those involving noise injection (like masking), is critical for enabling the Transformer to learn meaningful representations of vehicle interactions and distributions.
  • Robustness through Noise: Introducing noise into the training data helps the model generalize better and become more robust to variations and missing information in real-world traffic scenarios.
  • Decomposition of Traffic Tasks: The authors highlight the benefit of decomposing complex traffic analysis into sub-goals (distribution and temporal aspects) and designing pre-training tasks that address these individually.

Key Results

  • Spatial Distribution: The pre-trained model achieves excellent performance in capturing the spatial distribution of vehicles, with virtually no instances of vehicle overlap in the test set.
  • Position Accuracy: The RMSE for vehicle positions is 0.6059m, and the average Intersection over Union (IoU) for vehicle bounding boxes is 0.56.
  • Speed Prediction: Approximately 95% of predicted trajectories' speeds align closely with true speeds, with a deviation of 7.5144 m/s.
  • Fine-tuning Performance: After fine-tuning for trajectory compensation, the RMSE further decreased to 0.3202, and the IoU improved to 0.73, demonstrating the value of the pre-trained features.
  • Stability: The model exhibits remarkable stability, capable of continuously predicting trajectories for extended periods (200 frames) without significant error amplification or trajectory overlaps.

Limitations and Future Work

  • Computational Resources: Training large-scale Transformer models is computationally intensive. The authors acknowledge that their current GPU memory (8 x 32GB) limits the ability to train truly large models and incorporate longer input sequences.
  • Sequence Length: The current model uses a sequence length of 20 frames. The authors suggest that increasing this length could lead to more refined models and improved continuous prediction stability, but this would require more substantial computational resources and larger datasets.
  • Downstream Task Expansion: Future work includes extending the pre-training model to handle richer features and more complex downstream tasks, such as traffic control and simulating regional traffic patterns. They also plan to investigate biases in long-term trajectory predictions and evaluate the robustness of group simulations against classical car-following models.

Summary updated

Abstract

Understanding trajectory diversity is a fundamental aspect of addressing practical traffic tasks. However, capturing the diversity of trajectories presents challenges, particularly with traditional machine learning and recurrent neural networks due to the requirement of large-scale parameters. The emerging Transformer technology, renowned for its parallel computation capabilities enabling the utilization of models with hundreds of millions of parameters, offers a promising solution. In this study, we apply the Transformer architecture to traffic tasks, aiming to learn the diversity of trajectories within vehicle populations. We analyze the Transformer's attention mechanism and its adaptability to the goals of traffic tasks, and subsequently, design specific pre-training tasks. To achieve this, we create a data structure tailored to the attention mechanism and introduce a set of noises that correspond to spatio-temporal demands, which are incorporated into the structured data during the pre-training process. The designed pre-training model demonstrates excellent performance in capturing the spatial distribution of the vehicle population, with no instances of vehicle overlap and an RMSE of 0.6059 when compared to the ground truth values. In the context of time series prediction, approximately 95% of the predicted trajectories' speeds closely align with the true speeds, within a deviation of 7.5144m/s. Furthermore, in the stability test, the model exhibits robustness by continuously predicting a time series ten times longer than the input sequence, delivering smooth trajectories and showcasing diverse driving behaviors. The pre-trained model also provides a good basis for downstream fine-tuning tasks. The number of parameters of our model is over 50 million.
BibTeX
@misc{feng2023trtr,
  author = {Ruyi Feng and Zhibin Li and Bowen Liu and Yan Ding and Ou Zheng},
  title = {{TrTr: A Versatile Pre-Trained Large Traffic Model based on Transformer for Capturing Trajectory Diversity in Vehicle Population}},
  year = {2023},
  eprint = {2309.12677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.12677},
}
Figure