Abstract
Vehicle Trajectory Prediction (VTP) is a critical component of intelligent transportation systems. While much research focuses on autonomous driving from a vehicle's perspective, Surveillance VTP (SVTP) presents unique challenges, including high-angle viewpoints and noisy sensor data. This paper introduces VT-Former, a two-stage architecture combining Graph Isomorphism Networks (GIN) for social interaction modeling and a decoder-only Transformer for temporal trajectory prediction. Evaluated on the NGSIM and Carolina’s Highway Dataset (CHD), VT-Former achieves state-of-the-art performance, particularly in challenging real-world surveillance scenarios.
Introduction: The Shift to Surveillance VTP
Most contemporary VTP research is tailored for autonomous vehicles, utilizing onboard sensors to predict the paths of surrounding agents. However, there is a significant demand for Surveillance VTP (SVTP) in traffic management and accident prevention. SVTP systems rely on fixed cameras that often view traffic from high angles or eye-level, introducing perspective distortion and occlusion. Unlike the multi-modal predictions required for autonomous safety, surveillance systems typically prioritize a single, deterministic "most likely" path to optimize traffic flow.
The VT-Former Architecture
VT-Former addresses the SVTP problem through a specialized two-stage pipeline: Graph Attentive Tokenization (GAT) followed by a Transformer Predictor.
Graph Attentive Tokenization (GAT)
To model the complex social interactions between vehicles—such as following distances and lane-change reactions—VT-Former treats every vehicle in the scene as a node in a fully connected graph. It employs a Graph Isomorphism Network (GIN) to extract these social features. GINs are particularly effective at distinguishing different graph structures, allowing the model to better understand the spatial relationships between vehicles before converting them into tokens.
Transformer Predictor
The social tokens are processed by a decoder-only Transformer consisting of 8 layers and 4 attention heads. This architecture is optimized for low-latency, autoregressive prediction, generating the future trajectory ($T_{PH}$) one step at a time based on the observed history ($T_{OH}$) and previously predicted steps.
Experimental Setup and Datasets
The model was evaluated across three distinct perspectives to test its robustness:
- NGSIM (US-101 & I-80): A standard dataset providing clean, hand-annotated bird’s eye view data.
- Carolina’s Highway Dataset (CHD) - High-angle: Real-world footage from high-mounted surveillance cameras.
- Carolina’s Highway Dataset (CHD) - Eye-level: The most challenging perspective, featuring significant occlusion and machine-generated (noisy) annotations.
Results and the "Short History" Paradox
VT-Former demonstrated superior performance across the board. On the NGSIM dataset, it outperformed the previous state-of-the-art model, Pishgu, by 7.95% in Average Displacement Error (ADE) and 8.16% in Final Displacement Error (FDE).
A key finding of this study is the "Short History Paradox." In noisy surveillance environments like the CHD Eye-level dataset, using a shorter observation window (1 second) led to significantly better results than a longer window (3 seconds). Specifically, the short-history variant ($VT-Former_{SH}$) improved FDE by 37.3%. This suggests that in high-resolution surveillance, long histories may introduce chaotic noise that obscures immediate physical cues.
Conclusion
VT-Former provides a highly efficient (132k–155k parameters) and accurate solution for real-time highway surveillance. By leveraging GINs and Transformers, it effectively handles the unique noise and perspective challenges of SVTP. Future research will explore dynamic graph structures and neighborhood selection to further refine prediction accuracy and reduce computational overhead.
Summary updated