The Importance of Prior Knowledge in Precise Multimodal Prediction

Sergio Casas, Cole Gulino, Simon Suo, Raquel Urtasun

Abstract

Roads have well defined geometries, topologies, and traffic rules. While this has been widely exploited in motion planning methods to produce maneuvers that obey the law, little work has been devoted to utilize these priors in perception and motion forecasting methods. In this paper we propose to incorporate these structured priors as a loss function. In contrast to imposing hard constraints, this approach allows the model to handle non-compliant maneuvers when those happen in the real world. Safe motion planning is the end goal, and thus a probabilistic characterization of the possible future developments of the scene is key to choose the plan with the lowest expected cost. Towards this goal, we design a framework that leverages REINFORCE to incorporate non-differentiable priors over sample trajectories from a probabilistic model, thus optimizing the whole distribution. We demonstrate the effectiveness of our approach on real-world self-driving datasets containing complex road topologies and multi-agent interactions. Our motion forecasts not only exhibit better precision and map understanding, but most importantly result in safer motion plans taken by our self-driving vehicle. We emphasize that despite the importance of this evaluation, it has been often overlooked by previous perception and motion forecasting works.

Introduction

Predicting the future movement of surrounding actors is a critical "multimodal" challenge for autonomous vehicles. At any given moment, a vehicle might continue straight, turn, or stop. While modern deep learning models are proficient at capturing these various possibilities (high recall), they frequently generate unrealistic trajectories—such as drifting into sidewalks or teleporting across medians. This occurs because standard loss functions, like L2 distance, prioritize proximity to ground truth without considering the underlying "rules of the road."

Multimodal trajectory prediction in a complex driving scenario.

These unrealistic predictions pose significant challenges for the motion planner. If a model incorrectly predicts a high-probability swerve from an oncoming car, the self-driving vehicle (SDV) may react with unnecessary, jerky braking. Conversely, failing to predict a valid maneuver can lead to safety risks. This paper addresses the gap by integrating structured prior knowledge directly into the forecasting model's loss function.

Methodology

The proposed framework incorporates prior knowledge as a "soft constraint" during training, allowing the model to learn traffic rules while remaining resilient to actual rule-breaking behavior observed in real-world data.

Non-Differentiable Priors via REINFORCE

Rules such as "do not cross a solid yellow line" are discrete and non-differentiable, making them difficult to optimize using standard gradient descent. To overcome this, the authors employ the REINFORCE algorithm, a policy gradient method from reinforcement learning. This allows the road rules to be treated as a reward function, optimizing the entire probability distribution of predicted trajectories.

Structured Loss Functions

The framework introduces two primary structured losses:

  1. Reachable Lanes Loss: By building a lane graph that accounts for topology and traffic light states, the model penalizes predictions that enter unreachable lanes (e.g., turning the wrong way onto a one-way street) unless the ground truth demonstrates such a maneuver.
  2. SDV Route Loss: This loss focuses on precision for actors likely to interact with the SDV’s planned path. It aims to reduce false positives (which cause unnecessary braking) and false negatives (which pose safety risks).
Illustration of the proposed loss functions: Reachable Lanes Loss (left) and SDV Route Loss (right).

The base architecture utilizes SpAGNN, a Graph Neural Network that models multi-agent interactions and outputs a Mixture of Gaussians to represent multiple potential futures.

Evaluation and Results

The researchers evaluated the framework using the ATG4D and nuScenes datasets, which contain complex urban environments and diverse map specifications.

Key Insights

  • Precision for Planning: The authors argue that meanADE (average quality of all predictions) is more critical for motion planning than minADE (the best single prediction), as the planner must weigh the expected costs of all likely scenarios.
  • Planning-Aware Metrics: Evaluation focused on how predictions affect actual driving behavior, such as collision rates and passenger comfort (jerk).

Performance Metrics

  • Map Understanding: On the ATG4D dataset, the "Final Lane Error"—the frequency of predictions ending in impossible lanes—was reduced from 10% to 6% for vehicles moving straight.
  • Safety and Comfort: Integrating this model into a state-of-the-art motion planner resulted in a 17% reduction in collision rates. Additionally, the SDV exhibited lower lateral acceleration and jerk, providing a smoother and safer ride by eliminating "ghost" threats.

Conclusion and Future Work

This work demonstrates that baking "common sense" into motion forecasting significantly improves both the precision of predictions and the safety of autonomous driving. While the current focus is on vehicle actors, future research will look to extend these priors to pedestrians and bicyclists, who follow different social norms. Additionally, incorporating multi-agent collision avoidance as an explicit prior remains an area for further exploration.

Summary updated

Abstract

Roads have well defined geometries, topologies, and traffic rules. While this has been widely exploited in motion planning methods to produce maneuvers that obey the law, little work has been devoted to utilize these priors in perception and motion forecasting methods. In this paper we propose to incorporate these structured priors as a loss function. In contrast to imposing hard constraints, this approach allows the model to handle non-compliant maneuvers when those happen in the real world. Safe motion planning is the end goal, and thus a probabilistic characterization of the possible future developments of the scene is key to choose the plan with the lowest expected cost. Towards this goal, we design a framework that leverages REINFORCE to incorporate non-differentiable priors over sample trajectories from a probabilistic model, thus optimizing the whole distribution. We demonstrate the effectiveness of our approach on real-world self-driving datasets containing complex road topologies and multi-agent interactions. Our motion forecasts not only exhibit better precision and map understanding, but most importantly result in safer motion plans taken by our self-driving vehicle. We emphasize that despite the importance of this evaluation, it has been often overlooked by previous perception and motion forecasting works.
BibTeX
@inproceedings{casas2020importance,
  author = {Sergio Casas and Cole Gulino and Simon Suo and Raquel Urtasun},
  title = {{The Importance of Prior Knowledge in Precise Multimodal Prediction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=jFJ2417sjSM},
}
Figure