Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction

Mengshi Qi, Yuxin Yang, Huadong MaarXiv 2024 · DOI · arXiv

Introduction: The Challenge of Semantic Intentions

Predicting multi-agent trajectories is a fundamental task in autonomous systems, robotics, and sports analytics. While traditional models effectively capture physical constraints and agent-to-agent social distancing, they often overlook the high-level semantic intentions that drive movement in competitive environments. In professional basketball, for instance, a player's path is not merely a reaction to nearby obstacles but a deliberate execution of a team tactic, such as "Man-to-Man Defense" or a "Pick-and-Roll."

Tactical influence on basketball trajectories.

Without accounting for these group-level strategies, trajectory prediction models risk generating "unnatural" paths that ignore the strategic context of the scene. This paper introduces a novel framework designed to explicitly model these team-level strategies and agent intentions to produce more realistic and accurate forecasts.

The Tactic-e2e Framework

The proposed architecture, Tactic-e2e, integrates group-level tactical reasoning into a diffusion-based trajectory generation process. The framework consists of three primary modules:

Architecture of the Tactic-e2e model.
  1. Interaction Encoder & Denoising Module: Utilizing a conditional diffusion model, the system generates future trajectories through a series of denoising steps. To align these paths with specific team strategies, the authors employ classifier-free guidance, which effectively "nudges" the generated trajectories toward the expected tactical style (e.g., ensuring a defender sticks closer to an opponent during man-to-man defense).
  2. Semantic Intention Prediction: This module treats players and tactics as participants in a cooperative game. By calculating Banzhaf Interaction values, the model quantifies the influence of each agent on a specific tactic, providing a mathematically grounded way to identify which players are driving the group's behavior.
  3. Multi-Grained Feature Enhancement: To refine agent representations, the model combines global attention (capturing the entire scene) with local attention (focusing on critical objects like the ball).

Experimental Evaluation

The researchers evaluated Tactic-e2e on three major benchmarks:

  • Expanded NBA SportVU: A significantly enhanced version of the standard dataset, featuring human annotations for 16 distinct offensive and defensive tactics (e.g., Fast Break, Zone Defense).
  • TeamTrack-Soccer & TeamTrack-Basketball: These datasets were used to test the model's generalizability using "pseudo-labels" generated via unsupervised clustering to represent group-level movement patterns.

Results and Discussion

The Tactic-e2e model established new state-of-the-art (SOTA) performance across several metrics:

  • Trajectory Accuracy: On the NBA SportVU dataset, the model achieved a 9.5% improvement in Minimum Average Displacement Error (minADE) compared to previous SOTA methods like LED.
  • Tactic Prediction: The model reached a 60.23% Top-1 accuracy in predicting future team tactics, significantly outperforming both standard pooling methods (55.22%) and fine-tuned Large Language Models like Llama3-8B (38.17%).
  • Qualitative Realism: Visual analysis confirms that the model generates highly "interactive" paths. Defenders react to offensive shifts in a manner consistent with real-world basketball logic, avoiding the static or linear predictions common in older models.

Conclusion

By framing group interactions as a cooperative game and leveraging Banzhaf Interaction values, this research demonstrates that understanding the "why" behind movements is crucial for accurate trajectory prediction. While the current reliance on expert-annotated tactics presents a minor bottleneck, the success of unsupervised pseudo-labels suggests a path toward broader applicability in fields like autonomous driving and real-time sports broadcasting.

Summary updated

Abstract

Effective modeling of group interactions and dynamic semantic intentions is crucial for forecasting behaviors like trajectories or movements. In complex scenarios like sports, agents' trajectories are influenced by group interactions and intentions, including team strategies and opponent actions. To this end, we propose a novel diffusion-based trajectory prediction framework that integrates group-level interactions into a conditional diffusion model, enabling the generation of diverse trajectories aligned with specific group activity. To capture dynamic semantic intentions, we frame group interaction prediction as a cooperative game, using Banzhaf interaction to model cooperation trends. We then fuse semantic intentions with enhanced agent embeddings, which are refined through both global and local aggregation. Furthermore, we expand the NBA SportVU dataset by adding human annotations of team-level tactics for trajectory and tactic prediction tasks. Extensive experiments on three widely-adopted datasets demonstrate that our model outperforms state-of-the-art methods. Our source code and data are available at https://github.com/aurora-xin/Group2Int-trajectory.
BibTeX
@misc{qi2024learning,
  author = {Mengshi Qi and Yuxin Yang and Huadong Ma},
  title = {{Learning Group Interactions and Semantic Intentions for Multi-Object Trajectory Prediction}},
  year = {2024},
  eprint = {2412.15673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15673},
}
Figure