Abstract
Current trajectory forecasting models are typically evaluated using "marginal" metrics like Average Displacement Error (ADE), which allow models to select the best predictions for each agent independently across different future samples. This paper demonstrates that such metrics hide unrealistic behaviors, such as inter-agent collisions. By introducing and optimizing for joint metrics—which require all agents in a scene to be predicted accurately within the same future sample—the authors reveal a significant performance gap in existing models and achieve more socially compliant, collision-free multi-agent trajectories.
Introduction: The Marginal Metric Loophole
In autonomous driving and robotics, predicting the future trajectories of multiple agents (e.g., pedestrians or vehicles) is a multi-modal challenge. Standard evaluation protocols use "top-K" metrics, where a model generates $K$ possible future scenarios and is scored based on the best one.
However, standard metrics like Average Displacement Error (ADE) and Final Displacement Error (FDE) are "marginal." They calculate the minimum error for each agent independently. If Agent A’s best prediction occurs in Sample #3 and Agent B’s best prediction occurs in Sample #12, the metric "mixes and matches" these independent successes to produce an artificially low error score. This evaluation style fails to penalize "unnatural" joint behaviors, such as agents walking through each other, because the model is never forced to produce a single, coherent future for the entire scene.
Methods: Shifting to Joint Evaluation and Optimization
To address these shortcomings, the authors propose a shift toward joint evaluation and training:
- Joint Metrics (JADE and JFDE): Unlike ADE, Joint ADE (JADE) requires the model to select the single best sample for the entire scene. It averages the error across all agents within that specific sample before determining the minimum across all $K$ samples. This eliminates the "mix-and-match" loophole.
- Joint Optimization: The researchers modified state-of-the-art models, such as AgentFormer, by adding a "joint sample reconstruction loss." This loss penalizes the model if the collective positions of all agents in a single sample deviate from the ground truth.
- Collision Rate (CR): The study emphasizes the Collision Rate as a critical measure of social compliance, which marginal metrics often ignore.
Datasets
The methodology was validated on two primary pedestrian trajectory datasets:
- ETH / UCY: Five real-world scenes with diverse pedestrian interactions.
- Stanford Drone Dataset (SDD): A large-scale dataset featuring bird's-eye view trajectories across 20 campus scenes.
Key Insights and Results
The transition to joint metrics reveals that current models are less accurate than previously believed. The most striking finding is a "2x" performance gap: JADE scores are often twice as high as reported ADE scores, indicating that models struggle to predict coherent scenes.
By incorporating joint optimization, the authors achieved:
- Accuracy Improvements: A 7% improvement in JADE/JFDE on the ETH/UCY datasets compared to previous state-of-the-art results.
- Enhanced Safety: A 16% reduction in the mean collision rate.
- Emergent Social Awareness: Interestingly, forcing joint accuracy naturally reduced collisions without the need for explicit "collision-avoidance" rules in the loss function.
- Qualitative Realism: In dense environments, joint models better maintained group behaviors, preventing agents from diverging into physically impossible or socially awkward paths.
Limitations and Future Work
Joint optimization introduces additional constraints, making the learning process more complex. In simple scenes with minimal agent interaction, the benefits of joint loss are less pronounced and can occasionally lead to model confusion. Future research will focus on extending these joint metrics to more complex environments involving heterogeneous agents, such as mixed traffic with cars, cyclists, and pedestrians.
Summary updated