Abstract
We propose advances that address two key challenges in future trajectory prediction: (i) multimodality in both training data and predictions and (ii) constant time inference regardless of number of agents. Existing trajectory predictions are fundamentally limited by lack of diversity in training data, which is difficult to acquire with sufficient coverage of possible modes. Our first contribution is an automatic method to simulate diverse trajectories in the top-view. It uses pre-existing datasets and maps as initialization, mines existing trajectories to represent realistic driving behaviors and uses a multi-agent vehicle dynamics simulator to generate diverse new trajectories that cover various modes and are consistent with scene layout constraints. Our second contribution is a novel method that generates diverse predictions while accounting for scene semantics and multi-agent interactions, with constant-time inference independent of the number of agents. We propose a convLSTM with novel state pooling operations and losses to predict scene-consistent states of multiple agents in a single forward pass, along with a CVAE for diversity. We validate our proposed multi-agent trajectory prediction approach by training and testing on the proposed simulated dataset and existing real datasets of traffic scenes. In both cases, our approach outperforms SOTA methods by a large margin, highlighting the benefits of both our diverse dataset simulation and constant-time diverse trajectory prediction methods.
Introduction
Predicting the future trajectories of multiple agents is a critical task for autonomous vehicles. However, current approaches often struggle with three primary challenges: multimodality (the variety of possible future paths), data scarcity (the lack of diverse, complex maneuvers in standard datasets), and computational scaling (inference time that grows with the number of agents). SMART addresses these issues through a novel simulation-based data augmentation strategy and a grid-based model architecture that ensures constant-time inference.
Methodology
The SMART framework introduces two major innovations: a diverse simulation strategy and a fully convolutional model architecture.
1. Diverse Simulation Strategy
To overcome the limitations of "boring" real-world datasets where vehicles mostly drive straight, the authors developed a pipeline to generate diverse, scene-consistent trajectories. This method reconstructs real-world scenes from maps and mines existing datasets for realistic velocity profiles. These profiles are then used within a multi-agent simulator employing the Intelligent Driver Model (IDM) and MOBIL lane-changing model to create complex interactions.
2. The SMART Model Architecture
Unlike entity-based models that process agents individually, SMART utilizes a Bird’s-Eye-View (BEV) grid representation. By treating agents as features in a spatial map, the model can predict the future states of all agents simultaneously using fully convolutional operations.
The architecture features:
- Constant-Time Inference: The $O(1)$ complexity is achieved by processing the entire scene as a single grid, ensuring the system remains real-time regardless of the number of vehicles.
- State-Pooled ConvLSTMs: A novel pooling operation allows the model to maintain spatial and temporal consistency across multi-agent interactions without per-agent overhead.
- CVAE for Diversity: A Conditional Variational Autoencoder (CVAE) samples latent variables conditioned on maneuver labels (e.g., left turn, straight), enabling the generation of diverse, multimodal futures.
Experiments and Results
The model was evaluated on Argoverse and KITTI, alongside augmented versions (P-ArgoT and P-KITTI) containing simulated diverse behaviors. SMART demonstrated significant improvements in both efficiency and accuracy:
- Efficiency: SMART maintained a constant runtime of ~0.07s, significantly outperforming Social GAN as the number of agents increased.
- Accuracy: On the Argoverse dataset, SMART achieved a Final Displacement Error (FDE) of 2.47m, a substantial reduction compared to the 8.05m achieved by S-GAN.
- Scene Consistency: 90-95% of predicted paths were scene-consistent, staying within road boundaries, a much higher validity rate than traditional baselines.
Discussion and Conclusion
SMART provides a scalable and accurate solution for multi-agent trajectory prediction. By leveraging simulation to "teach" the model complex behaviors and employing a grid-based architecture for efficiency, it overcomes the primary hurdles of real-time autonomous navigation. Future work will focus on stricter adherence to traffic rules and explicit collision avoidance constraints to further improve safety in complex urban environments.
Summary updated