Collaborative Uncertainty in Multi-Agent Trajectory Forecasting

Bohan Tang, Yiqi Zhong, Ulrich Neumann, Gang Wang, Ya Zhang, Siheng Chen

Abstract

Uncertainty modeling is critical in trajectory forecasting systems for both interpretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajectories. However, the uncertainty brought by such correlations is neglected. To fill this gap, we propose a novel concept, collaborative uncertainty (CU), which models the uncertainty resulting from the interaction module. We build a general CU-based framework to make a prediction model learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module to current state-of-the-art (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. Extensive experiments on synthetic and large-scale benchmarks demonstrate that the CU-based framework helps SOTA systems improve their performance significantly, such as improving VectorNet by 57cm regarding Final Displacement Error on the nuScenes dataset.

1. Introduction

In autonomous driving, predicting the future movement of surrounding agents (vehicles, pedestrians, cyclists) is a fundamental task. Modern deep learning models utilize "interaction modules" to capture how agents influence one another—for instance, how a car slows down when a pedestrian enters a crosswalk. While these modules improve prediction accuracy, they introduce a hidden side effect: correlated uncertainty.

Most current systems treat each agent's prediction error as independent, effectively using a diagonal covariance matrix. However, if two agents are interacting, a mistake in predicting one agent's path is likely linked to a mistake in predicting the other's. Ignoring these correlations makes the model's confidence estimates unreliable and leaves potential performance gains on the table. This paper introduces Collaborative Uncertainty (CU) to model these correlated prediction errors.

2. Methodology: The CU Framework

Comparison of trajectory forecasting architectures: (a) standard encoder, (b) previous independent decoders, and (c) the proposed Collaborative Uncertainty (CU) framework.

The proposed framework moves beyond the standard assumption of independent agent uncertainties by estimating both Individual Uncertainty (IU) and Collaborative Uncertainty (CU).

Full Covariance Modeling

Instead of outputting a diagonal matrix, the model outputs a full covariance matrix. The diagonal elements ($\sigma_{ii}$) represent individual variance, while the off-diagonal elements ($\sigma_{ij}$) capture the "collaborative" correlations between agents. This allows the system to model the joint probability distribution $p(Y|X)$ of all agents simultaneously.

Distribution Flexibility and Stability

The framework supports both Multivariate Gaussian and Multivariate Laplace distributions. The Laplace version is particularly robust to outliers common in real-world traffic data. To ensure numerical stability and efficiency during training, the authors employ a square-root-free Cholesky decomposition. This "Cholesky trick" allows the network to predict the components of the inverse covariance matrix directly, avoiding the computational cost of inverting large matrices.

Plugin Architecture

A key strength of the CU framework is its design as a "plugin." It can be added to existing high-performing encoders like LaneGCN or VectorNet without requiring a total redesign of the underlying architecture.

3. Experimental Evaluation

The researchers evaluated their method on two massive real-world benchmarks: Argoverse (30,000+ scenarios) and nuScenes (complex urban scenes from Boston and Singapore), as well as synthetic datasets where ground-truth joint distributions were known.

Key Results

The addition of Collaborative Uncertainty consistently boosted the performance of SOTA models:

  • VectorNet on nuScenes: Achieved a massive 57 cm reduction in Final Displacement Error (FDE).
  • LaneGCN on Argoverse: Improved both Average Displacement Error (ADE) and FDE, outperforming the official SOTA implementation.
  • Synthetic Validation: On toy problems, the CU-based framework closely reconstructed the ground-truth joint distribution, whereas traditional "Individual Uncertainty" models failed to capture the correlations.

4. Discussion and Insights

Experimental results yielded several critical insights into multi-agent dynamics:

  1. Interaction Drives Uncertainty: The value of CU is highly sensitive to agent interaction. When agents are far apart, CU is near zero; when they approach intersections or drive side-by-side, CU spikes.
  2. Accuracy through Uncertainty: Modeling uncertainty actually helps the model learn a more accurate mean prediction. By correctly weighting the loss based on correlation, the model converges to a more precise trajectory.
  3. The Laplace Advantage: The Laplace-based CU framework generally outperformed the Gaussian version, confirming that real-world driving data often contains "heavy tails" (rare, extreme events) that Gaussian models struggle to handle.

5. Conclusion

Collaborative Uncertainty represents a significant step forward in making trajectory forecasting more reliable and accurate. While this work focuses on single-future prediction, extending CU to multi-modal (multi-future) forecasting is a critical next step. As autonomous systems scale to more crowded environments, modeling the joint uncertainty of all agents will be essential for safe navigation.

Summary updated

Abstract

Uncertainty modeling is critical in trajectory forecasting systems for both interpretation and safety reasons. To better predict the future trajectories of multiple agents, recent works have introduced interaction modules to capture interactions among agents. This approach leads to correlations among the predicted trajectories. However, the uncertainty brought by such correlations is neglected. To fill this gap, we propose a novel concept, collaborative uncertainty(CU), which models the uncertainty resulting from the interaction module. We build a general CU-based framework to make a prediction model to learn the future trajectory and the corresponding uncertainty. The CU-based framework is integrated as a plugin module to current state-of-the-art (SOTA) systems and deployed in two special cases based on multivariate Gaussian and Laplace distributions. In each case, we conduct extensive experiments on two synthetic datasets and two public, large-scale benchmarks of trajectory forecasting. The results are promising: 1) The results of synthetic datasets show that CU-based framework allows the model to appropriately approximate the ground-truth distribution. 2) The results of trajectory forecasting benchmarks demonstrate that the CU-based framework steadily helps SOTA systems improve their performances. Especially, the proposed CU-based framework helps VectorNet improve by 57cm regarding Final Displacement Error on nuScenes dataset. 3) The visualization results of CU illustrate that the value of CU is highly related to the amount of the interactive information among agents.
BibTeX
@inproceedings{tang2021collaborative,
  author = {Bohan Tang and Yiqi Zhong and Ulrich Neumann and Gang Wang and Ya Zhang and Siheng Chen},
  title = {{Collaborative Uncertainty in Multi-Agent Trajectory Forecasting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=vWXqguyKQc},
}
Figure