Joint Metrics Matter: A Better Standard for Trajectory Forecasting

Erica Weng, Hana Hoshino, Deva Ramanan, Kris Kitani

Abstract

Current trajectory forecasting models are typically evaluated using "marginal" metrics like Average Displacement Error (ADE), which allow models to select the best predictions for each agent independently across different future samples. This paper demonstrates that such metrics hide unrealistic behaviors, such as inter-agent collisions. By introducing and optimizing for joint metrics—which require all agents in a scene to be predicted accurately within the same future sample—the authors reveal a significant performance gap in existing models and achieve more socially compliant, collision-free multi-agent trajectories.

Introduction: The Marginal Metric Loophole

In autonomous driving and robotics, predicting the future trajectories of multiple agents (e.g., pedestrians or vehicles) is a multi-modal challenge. Standard evaluation protocols use "top-K" metrics, where a model generates $K$ possible future scenarios and is scored based on the best one.

However, standard metrics like Average Displacement Error (ADE) and Final Displacement Error (FDE) are "marginal." They calculate the minimum error for each agent independently. If Agent A’s best prediction occurs in Sample #3 and Agent B’s best prediction occurs in Sample #12, the metric "mixes and matches" these independent successes to produce an artificially low error score. This evaluation style fails to penalize "unnatural" joint behaviors, such as agents walking through each other, because the model is never forced to produce a single, coherent future for the entire scene.

Comparison of baseline marginal optimization vs. joint optimization. Marginal metrics allow for physically impossible collisions by mixing independent agent predictions.

Methods: Shifting to Joint Evaluation and Optimization

To address these shortcomings, the authors propose a shift toward joint evaluation and training:

  1. Joint Metrics (JADE and JFDE): Unlike ADE, Joint ADE (JADE) requires the model to select the single best sample for the entire scene. It averages the error across all agents within that specific sample before determining the minimum across all $K$ samples. This eliminates the "mix-and-match" loophole.
  2. Joint Optimization: The researchers modified state-of-the-art models, such as AgentFormer, by adding a "joint sample reconstruction loss." This loss penalizes the model if the collective positions of all agents in a single sample deviate from the ground truth.
  3. Collision Rate (CR): The study emphasizes the Collision Rate as a critical measure of social compliance, which marginal metrics often ignore.
Evaluation of three different future samples using JADE. Sample 3 provides the lowest joint error (JADE: 0.33) by accurately predicting multiple agents simultaneously.

Datasets

The methodology was validated on two primary pedestrian trajectory datasets:

  • ETH / UCY: Five real-world scenes with diverse pedestrian interactions.
  • Stanford Drone Dataset (SDD): A large-scale dataset featuring bird's-eye view trajectories across 20 campus scenes.

Key Insights and Results

The transition to joint metrics reveals that current models are less accurate than previously believed. The most striking finding is a "2x" performance gap: JADE scores are often twice as high as reported ADE scores, indicating that models struggle to predict coherent scenes.

By incorporating joint optimization, the authors achieved:

  • Accuracy Improvements: A 7% improvement in JADE/JFDE on the ETH/UCY datasets compared to previous state-of-the-art results.
  • Enhanced Safety: A 16% reduction in the mean collision rate.
  • Emergent Social Awareness: Interestingly, forcing joint accuracy naturally reduced collisions without the need for explicit "collision-avoidance" rules in the loss function.
  • Qualitative Realism: In dense environments, joint models better maintained group behaviors, preventing agents from diverging into physically impossible or socially awkward paths.

Limitations and Future Work

Joint optimization introduces additional constraints, making the learning process more complex. In simple scenes with minimal agent interaction, the benefits of joint loss are less pronounced and can occasionally lead to model confusion. Future research will focus on extending these joint metrics to more complex environments involving heterogeneous agents, such as mixed traffic with cars, cyclists, and pedestrians.

Summary updated

Abstract

Multi-modal trajectory forecasting methods commonly evaluate using single-agent metrics (marginal metrics), such as minimum Average Displacement Error (ADE) and Final Displacement Error (FDE), which fail to capture joint performance of multiple interacting agents. Only focusing on marginal metrics can lead to unnatural predictions, such as colliding trajectories or diverging trajectories for people who are clearly walking together as a group. Consequently, methods optimized for marginal metrics lead to overly-optimistic estimations of performance, which is detrimental to progress in trajectory forecasting research. In response to the limitations of marginal metrics, we present the first comprehensive evaluation of state-of-the-art (SOTA) trajectory forecasting methods with respect to multi-agent metrics (joint metrics): JADE, JFDE, and collision rate. We demonstrate the importance of joint metrics as opposed to marginal metrics with quantitative evidence and qualitative examples drawn from the ETH / UCY and Stanford Drone datasets. We introduce a new loss function incorporating joint metrics that, when applied to a SOTA trajectory forecasting method, achieves a 7\% improvement in JADE / JFDE on the ETH / UCY datasets with respect to the previous SOTA. Our results also indicate that optimizing for joint metrics naturally leads to an improvement in interaction modeling, as evidenced by a 16\% decrease in mean collision rate on the ETH / UCY datasets with respect to the previous SOTA. Code is available at \texttt{\hyperlink{https://github.com/ericaweng/joint-metrics-matter}{github.com/ericaweng/joint-metrics-matter}}.
BibTeX
@inproceedings{weng2023joint,
  author = {Erica Weng and Hana Hoshino and Deva Ramanan and Kris Kitani},
  title = {{Joint Metrics Matter: A Better Standard for Trajectory Forecasting}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01858},
}
Figure