Event2Tracking: Reconstructing Multi-Agent Soccer Trajectories Using Long-Term Multimodal Context

Harry Hughes, Michael Horton, Xinyu Wei, Harshala Gammulle, Clinton Fookes, Sridha Sridharan, Patrick LuceyAAAI 2025 · DOI

TL;DR

This paper introduces Event2Tracking, a novel method for reconstructing the precise movements of soccer players and the ball from noisy broadcast footage. By integrating long-term trajectory data (up to 60 seconds) with semantic event information (like passes and shots), Event2Tracking significantly improves accuracy, especially for occluded players and the ball, outperforming existing state-of-the-art methods.

What Problem Does This Paper Address?

Professional soccer games generate rich, multi-agent interaction data, making them an ideal testbed for studying complex systems. However, reconstructing accurate player and ball trajectories from broadcast footage is challenging due to several factors:

  • Occlusions: Players frequently exit and re-enter the camera's view, leading to significant periods of occlusion where their positions are unknown. Unlike other environments (e.g., pedestrian tracking), soccer's dynamic nature means occluded players are still actively participating and influencing the game.
  • Ball Tracking Difficulty: The ball is small, moves rapidly, is often occluded by players, and can be visually similar to pitch markings, making it exceptionally hard to track continuously and accurately from broadcast feeds.
  • Limited Context in Prior Work: Existing methods often rely on short-term trajectory data (≤10 seconds) and typically use only trajectory information (unimodal). This is insufficient for capturing the long-term dynamics and complex interactions that define a soccer match, especially when inferring the ball's position solely from player movements during heavy occlusions.

This paper aims to address these limitations by developing a model that leverages both extended temporal context and supplementary event data to achieve more robust and accurate trajectory reconstruction.

Methods

The core of the Event2Tracking approach lies in its long-term multimodal architecture.

  • Spatiotemporal Axial Attention: The model utilizes spatiotemporal axial attention, a technique that efficiently applies self-attention across both the temporal and spatial dimensions of multi-agent trajectories. This allows it to model individual agent movements over time (temporal attention) and their interactions within a given moment (spatial attention). A key advantage is its computational efficiency and permutation equivariance, meaning it can handle multi-agent data without needing to impose an artificial ordering on players. The paper extends the application of this technique to significantly longer time windows (60 seconds) than previously explored.
  • Multimodal Fusion: Event2Tracking fuses two primary data streams:
  • Broadcast Tracking Data: This is the noisy, often occluded trajectory data from broadcast footage, represented as a spatiotemporal grid.
  • Event Data: This is a sparse, temporal stream containing semantic information about game events (e.g., pass, shot, control), including their timestamp, location, and the involved player.
  • Event Encoder: A Transformer encoder processes the event data to create event embeddings.
  • Tracking Decoder: This component takes the broadcast tracking data and fuses it with the event embeddings using a Transformer-based architecture. It employs spatiotemporal axial attention for encoding trajectory dependencies and then uses temporal cross-attention to integrate the event context. This joint encoding allows the model to infer occluded positions and ball movements more accurately.
  • Temporal Localisation: To handle the non-uniform time intervals of both tracking and event data, the model calculates the elapsed time within each window and uses this for positional encoding, ensuring temporal consistency.

Datasets

The paper uses a large dataset comprising:

  • 700 professional soccer games for training.
  • 52 professional soccer games for evaluation.

Each game in the dataset includes paired data for:

  • Event data (mevent)
  • Broadcast tracking data (mbroadcast) - this data is characterized by occlusions and noise, with the ball's trajectory being fully occluded.
  • In-venue tracking data (min-venue) - this is the complete and accurate ground-truth trajectory data.

No new benchmarks were introduced; the focus was on evaluating against existing methods using this comprehensive dataset.

Key Insights and Ideas

  • Long-term context is crucial: The paper strongly demonstrates that extending the temporal window from short-term (≤10s) to longer-term (60s) significantly improves trajectory reconstruction accuracy, particularly for occluded agents. This is a critical insight for real-world applications where continuous tracking is desired.
  • Event data is a powerful signal: Integrating semantic event data (passes, shots, etc.) provides invaluable context that cannot be inferred from raw trajectories alone. This is especially impactful for reconstructing the ball's trajectory, which is otherwise highly ambiguous when players are occluded.
  • Spatiotemporal axial attention scales well: The authors show that spatiotemporal axial attention, known for its efficiency, is not only suitable for short-term but also for significantly longer trajectory modeling, making it a practical choice for complex, long-duration sequences.
  • Permutation Equivariance is essential: Soccer involves a variable number of agents with complex interactions. The use of permutation-equivariant models like spatiotemporal axial attention is vital to avoid combinatorial explosion and handle the inherent lack of a fixed spatial ordering among players.

Key Results

  • Superior Reconstruction Accuracy: Event2Tracking outperforms all baselines across various context lengths (10s to 60s) in terms of mean Average Displacement Error (mADE) for both players and the ball.
  • Ball Tracking Improvement: The method shows substantial gains in ball trajectory reconstruction, outperforming the next best model (STT) by 32-36% in mADE (ball) across context lengths. This is attributed to the effective use of event data.
  • Player Tracking Improvement: Event2Tracking achieves 3-11% lower mADE (players) compared to the STT baseline, reinforcing the benefit of multimodal context.
  • Performance Gains with Longer Context: The model's performance monotonically improves with longer context windows, with player mADE improving by 22% between 10s and 60s. This highlights the advantage of long-term modeling.
  • Qualitative Improvements: Visualizations show Event2Tracking's reconstructions more closely matching ground-truth, especially for occluded players and the ball, correctly inferring possession and realistic game actions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while their method significantly improves ball tracking, the ball's trajectory is still fully occluded in the broadcast tracking data, meaning its position is inferred.
  • The study focuses on deterministic baselines, acknowledging that real-world soccer trajectories are inherently stochastic.
  • The maximum context window considered was 60 seconds due to computational constraints.
  • Future Work:
    • Generative Models: Combining Event2Tracking with generative models to enable stochastic, diverse, and controllable generation of behaviors that are consistent with soccer's multimodal long-term structure.
  • General-Purpose Architecture: Adapting the model as a general architecture for detecting and predicting other team and player behaviors, such as the likelihood of scoring a goal within a specific time horizon.

Summary updated

BibTeX
@inproceedings{hughes2025event,
  author = {Harry Hughes and Michael Horton and Xinyu Wei and Harshala Gammulle and Clinton Fookes and Sridha Sridharan and Patrick Lucey},
  title = {{Event2Tracking: Reconstructing Multi-Agent Soccer Trajectories Using Long-Term Multimodal Context}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i11.33289},
}
Figure