Collaborative Uncertainty in Multi-Agent Trajectory Forecasting

Bohan Tang, Yiqi Zhong, Ulrich Neumann, Gang Wang, Siheng Chen, Ya Zhang

Introduction

While modern trajectory forecasting models use sophisticated modules to capture how agents interact on the road, they often ignore the collaborative uncertainty that arises from these interactions. This paper introduces a novel framework that models joint uncertainty across multiple agents. By treating this collaborative uncertainty as a "plugin," the framework significantly boosts the accuracy and reliability of state-of-the-art predictors like VectorNet and LaneGCN.

The Problem with Independent Uncertainty

Predicting where a car or pedestrian will move next is challenging because agents do not move in a vacuum—they constantly interact. If a car in the left lane suddenly brakes, the car behind it is likely to slow down or swerve.

Current deep learning models use "interaction modules" (such as Graph Neural Networks) to capture these dependencies and improve prediction accuracy. However, when it comes to estimating uncertainty—the model's confidence in its prediction—most systems treat each agent as an isolated island. They calculate "individual uncertainty" (the variance of a single agent's path) but ignore the correlations between agents.

This mismatch between a collaborative prediction process and an independent uncertainty estimate leads to sub-optimal performance and potentially dangerous overconfidence in complex traffic scenarios.

Comparison of previous independent decoders and the proposed Collaborative Uncertainty framework.

The Collaborative Uncertainty (CU) Framework

To bridge this gap, the authors propose a framework for Collaborative Uncertainty (CU). Instead of assuming the uncertainty of multiple agents is a diagonal matrix (where agents are strictly independent), they model it as a full covariance matrix.

Key technical highlights of the approach include:

  • Joint Probabilistic Modeling: The model outputs both the predicted trajectory (the mean) and the joint uncertainty (the covariance). The off-diagonal elements of this matrix explicitly represent the "Collaborative Uncertainty" between different agents.
  • Gaussian & Laplace Variants: The authors derive specific loss functions for two probability distributions. While Gaussian is standard, they demonstrate that a Multivariate Laplace distribution is more robust to the "outliers" and heavy-tailed behaviors often found in real-world driving data.
  • Numerical Stability: Estimating a full covariance matrix can be computationally expensive and mathematically unstable. To solve this, the authors use a square-root-free Cholesky decomposition, ensuring the matrix remains valid and the math stays efficient during training.
  • Plugin Architecture: The CU module is designed to be seamlessly added to existing "backbone" models (like VectorNet or LaneGCN) without requiring a total redesign of the underlying architecture.

Datasets and Evaluation

The researchers evaluated their method across both controlled synthetic environments and massive real-world benchmarks:

  1. Synthetic Datasets: Two self-generated datasets (Ternary Gaussian and Laplace) involving three interacting agents were used to verify if the model could accurately reconstruct a known "ground truth" distribution.
  2. Argoverse: A massive real-world benchmark with over 30,000 scenarios collected in Pittsburgh and Miami.
  3. nuScenes: A large-scale dataset featuring 1,000 scenes from Boston and Singapore, known for its highly complex urban environments.

Key Insights and Results

The addition of the CU module consistently improved performance across all tested benchmarks, yielding several important insights:

  • Interaction Causes Correlation: The authors demonstrate that using an interaction module (like an Agent-to-Agent GNN) inherently creates collaborative uncertainty. If the interaction module is removed, the benefit of CU modeling drops significantly.
  • Uncertainty as Information: Visualizations show that CU values spike in high-interaction scenarios—such as agents merging onto the same road or approaching a green light together. This suggests CU is not just a "safety buffer," but a meaningful representation of interactive information.
  • The Laplace Advantage: The Laplace-based CU framework generally outperforms the Gaussian version, better capturing the rare, sudden maneuvers typical of real-world driving.
  • State-of-the-Art Performance:
    • On nuScenes, adding the CU framework to VectorNet improved the Final Displacement Error (FDE) by a staggering 57 cm.
  • On Argoverse, the LaneGCN + CU implementation outperformed standard SOTA models, reducing FDE from 3.78m to 3.74m and Average Displacement Error (ADE) to 1.70m.
  • In synthetic tests, CU-enabled models approximated the ground-truth probability distribution with much lower KL-divergence compared to models limited to individual uncertainty.

Limitations and Future Work

The primary limitation of this work is its focus on single-future prediction (predicting the single most likely path). In reality, trajectory forecasting is often multi-modal (e.g., a car might turn left or go straight). Extending Collaborative Uncertainty to multi-future prediction is the logical next step.

Additionally, while the "plugin" nature of the framework is a major strength, calculating full covariance matrices for a very large number of agents simultaneously introduces computational overhead, which could become a bottleneck for real-time embedded systems in autonomous vehicles.

Summary updated

Abstract

Uncertainty modeling is critical in trajectory-forecasting systems for both interpretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajectories. However, the uncertainty brought by such correlations is neglected. To fill this gap, we propose a novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from the interaction module. We build a general CU-based framework to make a prediction model learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module to current state-of-the-art (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. In each case, we conduct extensive experiments on two synthetic datasets and two public, large-scale benchmarks of trajectory forecasting. The results are promising: 1) The results of synthetic datasets show that CU-based framework allows the model to nicely rebuild the ground-truth distribution. 2) The results of trajectory forecasting benchmarks demonstrate that the CU-based framework steadily helps SOTA systems improve their performances. Specially, the proposed CU-based framework helps VectorNet improve by 57 cm regarding Final Displacement Error on nuScenes dataset. 3) The visualization results of CU illustrate that the value of CU is highly related to the amount of the interactive information among agents.
BibTeX
@inproceedings{tang2021collaborative,
  author = {Bohan Tang and Yiqi Zhong and Ulrich Neumann and Gang Wang and Siheng Chen and Ya Zhang},
  title = {{Collaborative Uncertainty in Multi-Agent Trajectory Forecasting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=vWXqguyKQc},
}
Figure