The Importance of Prior Knowledge in Precise Multimodal Prediction

Sergio Casas, Cole Gulino, Simon Suo, Raquel Urtasun

TL;DR

This paper introduces a framework to bake "common sense" road rules—like staying in lanes and obeying traffic lights—directly into self-driving motion forecasting models. By using the REINFORCE gradient estimator to handle non-differentiable priors, the authors produce more realistic, precise predictions that significantly reduce "ghost" threats and lead to safer, smoother autonomous driving.

What Problem Does This Paper Address?

Predicting where other cars will go is a "multimodal" problem: at an intersection, a car might go straight, turn left, or stop. Current deep learning models are great at "covering" these possibilities (high recall), but they often produce "unrealistic" trajectories—like a car drifting into a sidewalk or teleporting across a median—because their loss functions only care about being close to the ground truth, not about the rules of the road.

These unrealistic predictions are a nightmare for the motion planner (the "brain" that decides how the self-driving car moves). If a model predicts a 5% chance that an oncoming car will suddenly swerve into your lane, the self-driving car (SDV) might slam on the brakes unnecessarily. Conversely, if it fails to predict a likely cut-in, the SDV might accelerate into a collision. The gap is that standard metrics (like L2 distance) treat all errors equally, whereas in the real world, some errors are harmless and others are catastrophic.

Methods

The authors propose a flexible framework to incorporate prior knowledge as a loss function during training.

  1. Non-Differentiable Priors: Rules like "don't cross a solid yellow line" are discrete and hard to differentiate mathematically. To solve this, the authors use the REINFORCE algorithm (a policy gradient method from Reinforcement Learning). This allows them to treat the road rules as a "reward" function and optimize the entire probability distribution of predicted paths.
  2. Reachable Lanes Loss: They build a "lane graph" that accounts for topology and current traffic light states. The model is penalized if it predicts a car entering a lane that is unreachable (e.g., turning into a one-way street the wrong way) unless the ground truth shows the car actually doing it. This ensures the model can still learn to predict "bad" behavior if it actually happens.
  3. SDV Route Loss: Not all actors are equally important. This loss prioritizes precision for cars that are likely to interact with the SDV’s planned path, reducing false positives (which cause jerky braking) and false negatives (which cause safety risks).
  4. Base Architecture: They build upon SpAGNN, a Graph Neural Network that models interactions between actors, and output a Mixture of Gaussians to represent multiple possible futures.

Datasets

The researchers evaluated their approach on two major datasets:

  • ATG4D: A massive, real-world self-driving dataset from Uber containing complex urban environments and multi-agent interactions.
  • nuScenes: A popular public large-scale dataset for autonomous driving, used here to prove the method's robustness across different map specifications.

Key Insights and Ideas

  • Precision > Recall for Planning: Most research focuses on minADE (how close the single best prediction is to the truth). This paper argues that meanADE (the average quality of all predictions) is more important for a motion planner that needs to weigh the expected cost of different scenarios.
  • Soft Constraints vs. Hard Constraints: Instead of forcing the model to never violate a rule (hard constraint), they use a loss function (soft constraint). This allows the model to remain "resilient" to rule-breakers—if a car is actually swerving, the data will override the prior.
  • Planning-Aware Metrics: The authors advocate for evaluating prediction models based on how they affect the driving (e.g., collision rates and jerk) rather than just geometric error.

Key Results

  • Improved Map Understanding: On the ATG4D dataset, the "Final Lane Error" (how often a prediction ends up in an impossible lane) dropped from roughly 10% to 6% for straight-moving vehicles.
  • Safer Driving: When plugged into a state-of-the-art motion planner, the proposed method reduced the SDV's collision rate by 17% compared to the strongest baseline.
  • Smoother Rides: Because the model produced fewer "unrealistic" threats, the SDV exhibited lower lateral acceleration and jerk, leading to a more comfortable experience for passengers without sacrificing progress.

Limitations and Future Work

  • Actor Types: The current study focuses primarily on vehicles. The authors note that extending these priors to pedestrians and bicyclists (who follow different "social" rules rather than strict lane geometry) is a necessary next step.
  • Inter-agent Priors: While the model understands the map, it doesn't explicitly use a prior for "cars don't drive through each other." Incorporating multi-agent collision avoidance as a prior is suggested for future work.
  • Computational Overhead: Using REINFORCE requires drawing many samples during training, which can be more computationally expensive than standard supervised learning.

Summary updated

Abstract

Roads have well defined geometries, topologies, and traffic rules. While this has been widely exploited in motion planning methods to produce maneuvers that obey the law, little work has been devoted to utilize these priors in perception and motion forecasting methods. In this paper we propose to incorporate these structured priors as a loss function. In contrast to imposing hard constraints, this approach allows the model to handle non-compliant maneuvers when those happen in the real world. Safe motion planning is the end goal, and thus a probabilistic characterization of the possible future developments of the scene is key to choose the plan with the lowest expected cost. Towards this goal, we design a framework that leverages REINFORCE to incorporate non-differentiable priors over sample trajectories from a probabilistic model, thus optimizing the whole distribution. We demonstrate the effectiveness of our approach on real-world self-driving datasets containing complex road topologies and multi-agent interactions. Our motion forecasts not only exhibit better precision and map understanding, but most importantly result in safer motion plans taken by our self-driving vehicle. We emphasize that despite the importance of this evaluation, it has been often overlooked by previous perception and motion forecasting works.
BibTeX
@inproceedings{casas2020importance,
  author = {Sergio Casas and Cole Gulino and Simon Suo and Raquel Urtasun},
  title = {{The Importance of Prior Knowledge in Precise Multimodal Prediction}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=jFJ2417sjSM},
}

Loading related content...

Figure