SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction

Sriram N. N, Buyu Liu, Francesco Pittaluga, Manmohan Chandraker

Abstract

Predicting the future trajectories of multiple interacting agents is a critical task for autonomous driving. Current methods often suffer from two primary limitations: computational complexity that scales linearly with the number of agents ($O(n)$) and a lack of diversity in training data, which is typically dominated by simple, straight-line driving. We present SMART, a framework that addresses these challenges through a novel scene-centric architecture and an automated simulation pipeline for diverse behavior generation. SMART achieves constant-time inference regardless of agent count and significantly improves prediction accuracy for complex maneuvers.

Introduction

Autonomous vehicles must navigate complex environments where the future actions of surrounding drivers are uncertain. To ensure safety, a prediction system must be both fast and capable of representing multiple possible futures (multimodality). However, traditional agent-centric models process each vehicle individually, leading to latency issues in crowded scenes. Furthermore, real-world datasets often lack sufficient examples of rare but critical maneuvers, such as sharp turns or aggressive lane changes. SMART overcomes these hurdles by treating the entire scene as a single grid and augmenting training data with realistic, simulated interactions.

The SMART Architecture

At the core of SMART is a scene-centric approach that utilizes a Bird’s-Eye-View (BEV) grid representation. Instead of modeling agents as isolated entities, the system processes the entire environment as a multi-layered image, allowing it to implicitly learn social interactions through convolutional operations.

The SMART architecture uses a conditional generator with a convolutional encoder-decoder structure to process multi-layered input maps, enabling constant-time trajectory prediction for all agents.

Key Innovations:

  • ConvLSTM with State Pooling: Grid-based models often struggle with "jitter" as agents move between cells. SMART introduces a state-pooling mechanism that transfers an agent's internal memory state to its new predicted location, ensuring temporal consistency.
  • Conditional Variational Autoencoder (CVAE): To capture multimodality, the model uses a CVAE to sample various potential behaviors (e.g., turning vs. going straight) from a latent space.
  • Constant-Time Inference: By processing the entire grid in a single forward pass, SMART maintains a fixed inference time (approx. 0.07s), making it highly suitable for real-time applications.

Diverse Behavior Simulation

To address the "straight-line bias" in existing datasets, the authors developed an automated simulation pipeline. This system mines real-world maps and trajectories to extract realistic velocity profiles and then applies advanced vehicle dynamics models to create new, challenging scenarios.

The simulation pipeline generates diverse training data by combining map data with maneuver identification and vehicle dynamics models like IDM and MOBIL.

The simulation utilizes the Intelligent Driver Model (IDM) for longitudinal control and MOBIL for lane-changing logic. This allows for the generation of thousands of physically plausible scenarios—such as complex intersection crossings—that are consistent with road geometry and traffic rules. Training on this diverse data enables the model to better generalize to real-world edge cases.

Experimental Results

SMART was evaluated on major benchmarks, including Argoverse and KITTI, as well as augmented versions (P-ArgoT and P-KITTI) designed to test maneuver diversity.

  • Efficiency: While baseline models like Social GAN see processing times triple as the number of agents increases, SMART maintains a constant 0.07-second latency for up to 10 agents.
  • Accuracy: On the Argoverse dataset, SMART achieved a Final Displacement Error (FDE) of 1.02 meters at 5 seconds, a significant improvement over the 2.54 meters recorded by Social GAN.
  • Scene Compliance: Predictions generated by SMART are more likely to respect road boundaries and lane constraints compared to other generative models, which often produce trajectories that drift off-road.

Conclusion

SMART represents a significant step forward in multi-agent trajectory prediction. By shifting from an agent-centric to a scene-centric paradigm and leveraging high-fidelity simulation for data augmentation, the framework provides a scalable and robust solution for autonomous navigation. Future work will focus on integrating 3D perspectives and more explicit traffic rule constraints to further refine prediction reliability in complex urban environments.

Summary updated

Abstract

We propose advances that address two key challenges in future trajectory prediction: (i) multimodality in both training data and predictions and (ii) constant time inference regardless of number of agents. Existing trajectory predictions are fundamentally limited by lack of diversity in training data, which is difficult to acquire with sufficient coverage of possible modes. Our first contribution is an automatic method to simulate diverse trajectories in the top-view. It uses pre-existing datasets and maps as initialization, mines existing trajectories to represent realistic driving behaviors and uses a multi-agent vehicle dynamics simulator to generate diverse new trajectories that cover various modes and are consistent with scene layout constraints. Our second contribution is a novel method that generates diverse predictions while accounting for scene semantics and multi-agent interactions, with constant-time inference independent of the number of agents. We propose a convLSTM with novel state pooling operations and losses to predict scene-consistent states of multiple agents in a single forward pass, along with a CVAE for diversity. We validate our proposed multi-agent trajectory prediction approach by training and testing on the proposed simulated dataset and existing real datasets of traffic scenes. In both cases, our approach outperforms SOTA methods by a large margin, highlighting the benefits of both our diverse dataset simulation and constant-time diverse trajectory prediction methods.
BibTeX
@inproceedings{n2020smart,
  author = {Sriram N. N and Buyu Liu and Francesco Pittaluga and Manmohan Chandraker},
  title = {{SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction}},
  booktitle = {European Conference on Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=qdpcbBSQHE7},
}
Figure