Multi-Modal Trajectory Prediction of NBA Players

Sandro Hauri, Nemanja Djuric, Vladan Radosavljevic, Slobodan VuceticWACV 2021

TL;DR

This paper introduces a novel multi-modal deep learning model for predicting NBA player trajectories, acknowledging that players can consider multiple movement options. Their approach, trained with a specialized multi-modal loss function, significantly outperforms existing state-of-the-art methods in accuracy and realism, and can even learn individual player styles.

What Problem Does This Paper Address?

Predicting the movement of individuals in dynamic environments is a crucial challenge with applications ranging from autonomous driving to crowd simulation. In team sports like basketball, accurately predicting player trajectories is vital for tactical advantage, post-game analysis, and even optimizing broadcast views. Existing methods often struggle to capture the inherent uncertainty and multiple plausible future paths a skilled athlete might consider during a game. This paper specifically tackles the problem of predicting individual NBA player movements, recognizing that players don't just follow a single predetermined path but rather choose from a set of potential actions.

Methods

The core of this work is a Multi-modal Basketball Trajectories (MBT) model built upon a two-layer LSTM architecture. Unlike traditional models that predict a single trajectory, MBT is designed to output multiple predicted trajectories along with their associated probabilities. This multi-modality is crucial for capturing the diverse options available to players.

A key innovation is the multi-modal loss function used for training. This function combines a classification loss (to learn the probabilities of different modes) with a trajectory loss (to penalize errors in the predicted paths). To address the issue of "mode collapse" (where some predicted paths are never learned), they employ a relaxed Kronecker delta in the loss function, which gradually reduces the weight given to less likely modes during training.

The paper also explores different distance functions to determine which predicted trajectory is "closest" to the ground truth for training purposes, evaluating Mean Squared Error (MSE), displacement error, and velocity error. Furthermore, they propose a two-stage training approach: first, a general model is trained on all NBA players, and then this model is fine-tuned on individual player data to capture unique playing styles.

Datasets

The study utilizes a large dataset of publicly available movement data collected from 632 NBA games during the 2015-2016 season. This dataset comprises 114,294 offensive possessions, providing player and ball locations at a high temporal resolution (0.04s, downsampled to 0.12s). The data was split into training (90%) and testing (10%) sets. For per-player analysis, possessions featuring specific players were extracted, with examples like Stephen Curry having thousands of associated possessions.

Key Insights and Ideas

  • Multi-modality is essential for realistic player prediction: NBA players, as skilled experts, consider multiple movement options. A single-trajectory prediction model cannot capture this inherent uncertainty.
  • Relaxed Kronecker delta prevents mode collapse: Using a relaxed version of the Kronecker delta in the loss function is crucial for ensuring that all predicted modes are trained and contribute to the model's learning, preventing poorly initialized paths from being ignored.
  • Velocity prediction leads to more realistic trajectories: Predicting velocities rather than direct locations results in smoother and more physically plausible player movements, as evidenced by lower MSE scores.
  • Per-player fine-tuning captures individual styles: A general model provides a good baseline, but fine-tuning on individual player data significantly improves prediction accuracy and allows the model to learn distinct playing styles (e.g., a fast center vs. a defensive specialist).
  • Mode probabilities are well-calibrated: The model's predicted probabilities for each trajectory mode closely align with empirical probabilities, indicating a reliable understanding of which paths are more likely.

Key Results

The proposed MBT41 model (4 modes, velocity prediction, distance function (7)) significantly outperforms state-of-the-art methods like SocialGAN and MACRO VRNN across various error metrics (ADE, FDE, MSE) for prediction horizons of up to 40 time steps (approximately 4.8 seconds). For H=40, MBT41 achieved 24% and 21% smaller ADE than MACRO VRNN4 and SocialGAN4, respectively.

Qualitatively, the model generates distinct and plausible trajectories for each mode. The highest probability path closely matches the ground truth, while other paths offer diverse, realistic movement options. The paper also demonstrates that per-player fine-tuning leads to improved performance for specific players, capturing their unique tendencies (e.g., Stephen Curry's preference for 3-pointers). Furthermore, the predicted trajectories exhibit more realistic accelerations compared to baseline models, indicating a higher degree of physical plausibility.

Limitations and Future Work

The authors acknowledge that while their model captures individual player styles, it does not explicitly model opponent strategies or how players might react to an opponent's predicted movements. They also note that their approach currently focuses on individual player prediction rather than modeling the multi-modal behavior of the entire team in concert.

Future work includes exploring ways to model the collective behavior of teams and developing strategies to predict how players might counter opponent actions. They also suggest investigating the use of other features that might influence observed trajectories, such as game clock, home/away status, or foul calls, although they demonstrate that their current approach can implicitly capture some of this information through historical data.

Summary updated

BibTeX
@inproceedings{hauri2021multi,
  author = {Sandro Hauri and Nemanja Djuric and Vladan Radosavljevic and Slobodan Vucetic},
  title = {{Multi-Modal Trajectory Prediction of NBA Players}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure