Evaluation of Trajectory Distribution Predictions with Energy Score

Novin Shahroudi, Mihkel Lepson, Meelis KullICML 2024 · OpenReview

Introduction

In the world of autonomous driving and robotics, predicting where a pedestrian or vehicle will move next is a game of probabilities. Because the future is inherently uncertain, modern models do not just output one single path; they output a distribution of many plausible trajectories, a task known as Multimodal Trajectory Prediction.

Despite these models becoming increasingly sophisticated, the evaluation methods used to assess them have not kept pace. Trajectory prediction models are still largely evaluated using "Minimum of N" (MoN) metrics that ignore the overall predicted distribution. This paper demonstrates that MoN metrics are mathematically "improper"—meaning they can inadvertently reward incorrect models—and proposes the Energy Score as a strictly proper, distribution-aware alternative for more reliable evaluation.

The Flaw in "Minimum of N" Metrics

The industry-standard metrics for evaluating trajectory models are Minimum of N (MoN), which include minADE (Average Displacement Error) and minFDE (Final Displacement Error). These metrics are fundamentally flawed because they only look at the single best trajectory out of $N$ samples that happened to be closest to the ground truth.

The authors argue that this approach ignores the rest of the predicted distribution. A model could produce one lucky guess alongside 99 dangerous, highly inaccurate ones and still receive a "perfect" score. Mathematically, MoN is not a strictly proper scoring rule: it does not necessarily reach its minimum value when the predicted distribution perfectly matches the real-world distribution.

This creates a "Lucky Guess" problem. MoN metrics reward mode collapse or overconfidence. If a model places a very sharp peak in the wrong place, but you sample it enough times, you might eventually get a sample near the truth. MoN rewards this behavior, whereas a proper score would penalize the overall distributional mismatch.

1D Bimodal Toy Example comparing Ground Truth and Forecast distributions

Furthermore, MoN suffers from sample size sensitivity. The authors provide mathematical proofs showing that MoN values decrease simply by increasing the number of samples ($K$). This makes it impossible to cross-compare models evaluated with different sample sizes; a model's minADE with $K=100$ will almost always look better than one with $K=20$, regardless of actual model quality.

The Energy Score Solution

To resolve these evaluation flaws, the researchers apply the framework of Proper Scoring Rules to trajectory prediction, proposing the Energy Score (ES). The Energy Score is a multivariate generalization of the Continuous Ranked Probability Score (CRPS). Its formula balances two critical terms:

  1. Accuracy (Term 1): The expected distance between predicted samples and the ground truth.
  2. Diversity (Term 2): The expected distance between the predicted samples themselves.

To help researchers debug their models more granularly, the authors also introduce two diagnostic variations of the Energy Score:

  • EST (Temporal Energy Score): Marginalizes spatial data to focus strictly on timing errors.
  • ESS (Spatial Energy Score): Marginalizes temporal data to focus strictly on positioning errors.
Trajectory forecasts and their corresponding Temporal, Spatial, and overall Energy Scores

Experimental Validation & Key Results

The authors validated their claims across three distinct setups: toy 1D examples, a synthetic autoregressive process, and the standard ETH/UCY real-world benchmark for human trajectory prediction.

  • Propriety Showcase: In controlled simulations, the Energy Score was correctly minimized at a parameter deviation of 0 (the exact ground truth). In contrast, minFDE was minimized by biased parameters that produced higher variance than the ground truth, proving MoN's optimization bias.
  • Ranking Discrepancies: On the ETH/UCY benchmark, adopting the Energy Score completely changed the "leaderboard." While minADE/minFDE consistently ranked NPSN-based models as the best, the Energy Score favored Quasi-Monte Carlo (QMC) sampling in several scenarios. This suggests QMC produces a more faithful representation of uncertainty, whereas models trained specifically to minimize minADE may actually produce worse overall distributions.
  • Stability Across Sample Sizes: As the sample size $K$ increased from 20 to 100, the optimal ADE dropped by approximately 75%. Meanwhile, the Energy Score remained remarkably stable within a tight, consistent range (~11.7 to 11.2), proving its reliability across different testing conditions.

Limitations and Future Work

While the Energy Score offers a massive leap forward in evaluation rigor, it does come with computational trade-offs. Calculating the Energy Score requires an $O(K^2)$ operation (comparing all pairs of samples), which is more expensive than the $O(K)$ MoN calculation. However, the authors note that $O(K)$ approximations for the Energy Score exist and can be utilized.

Additionally, the authors acknowledge that in specific safety-critical edge cases, a downstream user might only care about the best-case or worst-case sample. They suggest that while the Energy Score should serve as the primary metric for overall distribution quality, MoN can remain a secondary metric for specific, task-aware requirements. Finally, the impact of the score's variance when $K$ is extremely low (e.g., < 10) warrants further investigation in future work.

Summary updated

Abstract

Predicting the future trajectory of surrounding objects is inherently uncertain and vital in the safe and reliable planning of autonomous systems such as in self-driving cars. Although trajectory prediction models have become increasingly sophisticated in dealing with the complexities of spatiotemporal data, the evaluation methods used to assess these models have not kept pace. "Minimum of N" is a common family of metrics used to assess the rich outputs of such models. We critically examine the Minimum of N within the proper scoring rules framework to show that it is not strictly proper and demonstrate how that could lead to a misleading assessment of multimodal trajectory predictions. As an alternative, we propose using Energy Score-based evaluation measures, leveraging their proven propriety for a more reliable evaluation of trajectory distribution predictions.
BibTeX
@inproceedings{shahroudi2024evaluation,
  author = {Novin Shahroudi and Mihkel Lepson and Meelis Kull},
  title = {{Evaluation of Trajectory Distribution Predictions with Energy Score}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=FCmWhJQ14I},
}
Figure