Unified Multi-Agent Trajectory Modeling with Masked Trajectory Diffusion

Songru Yang, Zhenwei Shi, Zhengxia ZouICCV 2025

TL;DR

This paper introduces UniMTD, a novel unified framework for multi-agent trajectory modeling that handles prediction, imputation, and recovery simultaneously. It achieves this by decoupling observed and unobserved trajectory segments using a unidirectional attention mechanism and a masked diffusion approach, leading to significant improvements in accuracy and fidelity across various datasets.

What Problem Does This Paper Address?

Understanding and predicting the movement of multiple agents in a scene is crucial for applications like autonomous driving, surveillance, and scene generation. Existing methods often struggle with real-world scenarios where agent observations are incomplete due to occlusions or sensor limitations. These methods typically fail to handle the combined tasks of trajectory prediction (forecasting future movement), imputation (filling in missing past or present segments), and recovery (reconstructing significantly occluded trajectories) within a single, unified framework. This limitation stems from a coupled approach that struggles to disentangle accurate observed data from uncertain, noisy, or missing data, leading to suboptimal performance.

Methods

UniMTD proposes a decoupled denoising diffusion modeling paradigm. The core innovations are:

  • Unidirectional Attention (UA): This acts as an "information valve" to control the flow of information between observed and masked (unobserved) trajectory segments. It ensures that information from observed segments influences the modeling of masked segments, while preventing noise from masked segments from corrupting the accurate representations of observed ones. This is achieved through unidirectional masking and a progressive removal process.
  • Unidirectional MoE (UMoE): This architecture leverages local and global experts to handle varying levels of missing observations. A complex adaptive router dynamically allocates experts based on the mask ratio, allowing for flexible modeling of different missing data scenarios.
  • Cached Diffusion Model (CDiff): To improve efficiency, CDiff reuses pre-calculated deterministic features from the encoder as a starting point for the diffusion process. This significantly reduces the number of denoising steps required, leading to faster inference with minimal impact on generation quality.

Datasets

The paper evaluates UniMTD on a variety of datasets:

  • Human Motion Datasets: Basketball-U, Soccer-U, and Football-U*.
  • Vehicle Motion Dataset: Vehicle-U (Easy/Ordinary/Hard)*.

These datasets are used to demonstrate the model's versatility across different domains and complexities of agent behavior. The mask strategy used for evaluation is consistent with prior work [71] for fair comparison.

Key Insights and Ideas

  • Decoupling is Key: The most significant insight is that explicitly separating observed (accurate) and unobserved (uncertain) trajectory information is crucial for robust multi-agent modeling, especially when dealing with significant data loss.
  • Unidirectional Flow for Control: The unidirectional attention mechanism effectively acts as a gatekeeper, allowing for controlled information exchange and preventing noise propagation.
  • Efficiency through Caching: Reusing encoder outputs in the diffusion decoder (CDiff) is a clever way to drastically reduce computational cost without sacrificing performance, a common bottleneck in diffusion models.
  • Unified Framework Benefits All Tasks: The results suggest that a unified approach not only handles prediction, imputation, and recovery effectively but also improves performance on individual tasks compared to specialized methods.

Key Results

UniMTD demonstrates state-of-the-art performance across multiple metrics:

  • Accuracy: Achieves a 74% improvement in minADE20 on human motion datasets and significant advantages across various metrics on vehicle datasets.
  • Fidelity: Shows substantial gains (e.g., 91% on OOB, 66% on Step, 69% on Path-L) indicating more realistic trajectory generation.
  • Efficiency: For a scenario with 11 agents, UniMTD takes 0.18s for inference, significantly faster than diffusion-based methods like SSSD (over 10s). CDiff reduces the number of denoising steps from 200 to 5.
  • Qualitative: UniMTD generates more accurate and faithful trajectories, particularly excelling in recovering complex arc and circular paths, outperforming competitors like UniTraj and SSSD.

Limitations and Future Work

  • Model Complexity: While UniMTD achieves strong results, the paper notes that increasing model complexity (e.g., number of experts) can lead to performance improvements in some metrics but deterioration in others, suggesting a trade-off.
  • Future Exploration: The authors suggest that UniMTD's success in unified trajectory modeling opens avenues for further exploration in embodied foundation models, hinting at broader applications beyond trajectory prediction. The paper also implicitly suggests that further tuning of expert configurations and router mechanisms within UMoE could yield even better results.

Summary updated

BibTeX
@inproceedings{yang2025unified,
  author = {Songru Yang and Zhenwei Shi and Zhengxia Zou},
  title = {{Unified Multi-Agent Trajectory Modeling with Masked Trajectory Diffusion}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure