MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, Benjamin Sapp

TL;DR

MultiPath++ is a state-of-the-art behavior prediction model for autonomous driving that moves away from traditional, computationally expensive image-based scene representations. By using sparse "polyline" encodings, a novel Multi-Context Gating (MCG) fusion mechanism, and learned trajectory anchors, it achieves top-tier performance on the Waymo and Argoverse benchmarks while remaining highly efficient.

What Problem Does This Paper Address?

Predicting where a car, cyclist, or pedestrian will go next is one of the hardest parts of autonomous driving. The environment is inherently stochastic (a driver might turn left or go straight at the last second) and heterogeneous (you have to fuse map data, traffic lights, and the movement history of dozens of nearby agents).

Earlier SOTA models, including the original MultiPath, often relied on "rasterization"—turning the world into a top-down image and using standard Computer Vision CNNs to process it. However, rasterization has major flaws: it’s computationally wasteful (most pixels are empty), it loses precision due to grid resolution, and it struggles to capture long-range interactions between distant road elements.

Methods

MultiPath++ introduces three major architectural upgrades to solve these issues:

  1. Sparse Encoding: Instead of images, the model represents the world as a collection of "polylines" (vectors describing lanes and boundaries) and raw state sequences (position, velocity, etc.) for agents. This is more compact and preserves high-precision geometric details.
  2. Multi-Context Gating (MCG): This is the model’s core fusion engine. Rather than using expensive "all-to-all" attention (where every element looks at every other element), MCG creates a summary "context vector" for the scene. This vector is then used to "gate" (filter and influence) the features of individual elements. It provides the benefits of cross-attention at a fraction of the computational cost.
  3. Learned Latent Anchors: The original MultiPath used fixed, pre-defined trajectory "anchors" (like a set of standard turns). MultiPath++ instead learns these anchors as latent embeddings during training. This allows the model to adapt its "templates" for future motion directly from the data.
  4. EM-based Ensembling: To combine predictions from multiple models, the authors use a greedy iterative clustering approach based on the Expectation-Maximization (EM) algorithm. This ensures the final output is a diverse, non-redundant set of possible futures.

Datasets

The model was evaluated on two of the industry's most competitive benchmarks:

  • Waymo Open Motion Dataset (WOMD): A massive dataset with 1.1 million examples and an 8-second prediction horizon.
  • Argoverse Motion Forecasting Dataset: A dataset focused on 3-second prediction horizons in urban environments.

Key Insights and Ideas

  • Efficiency over Density: The shift from dense rasters to sparse polylines proves that "seeing" the world as a set of structured objects is more effective for motion planning than treating it as a picture.
  • Contextual Awareness is Key: A lane line's "meaning" changes if there is a car currently driving on it. MCG allows the model to bake this relationship into the features early in the encoding process.
  • Diversity via Clustering: In behavior prediction, you don't want the "average" of two models (which might be a path into a wall); you want a representative set of both models' best guesses. The EM-clustering approach for ensembling preserves this vital multimodality.
  • Inductive Bias vs. Flexibility: While the authors experimented with "kinematic controls" (predicting steering and acceleration), they found that predicting raw coordinates still performed better on standard distance metrics, though it occasionally produced less "smooth" paths.

Key Results

MultiPath++ achieved dominant results at the time of publication:

  • Waymo Leaderboard: Ranked 1st overall. It achieved a Mean Average Precision (mAP) of 0.409 and a Minimum Average Distance Error (minADE) of 0.556m, significantly outperforming the original MultiPath (which had a minADE of 0.880m).
  • Argoverse Leaderboard: Ranked 4th overall, showing strong generalization across different sensor setups and map formats.
  • Ablation Studies: The researchers showed that stacking MCG blocks and using learned anchors provided the most significant boosts in accuracy and "miss rate" (how often the model completely fails to predict the true path).

Limitations and Future Work

  • Kinematic Feasibility: The model sometimes predicts trajectories that are physically impossible for a car to follow (e.g., turning too sharply). While they explored "control-based" outputs to fix this, it resulted in lower accuracy on public leaderboards.
  • Joint Prediction: The current model predicts trajectories for agents independently. The authors note that explicitly modeling the joint future of all agents (e.g., "if Car A yields, Car B will go") remains a computationally expensive frontier.
  • Sensitivity to Perception: Because the model relies on raw state estimates, it can be sensitive to errors or noise in the upstream perception and tracking systems.

Summary updated

Abstract

Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model that achieves state-of-the-art performance on popular benchmarks. MultiPath++ improves the MultiPath architecture [34] by revisiting many design choices. The first key design difference is a departure from dense image-based encoding of the input world state in favor of a sparse encoding of heterogeneous scene elements: MultiPath++ consumes compact and efficient polylines to describe road features, and raw agent state information directly (e.g., position, velocity, acceleration). We propose a context-aware fusion of these elements and develop a reusable multi-context gating fusion component. Second, we reconsider the choice of pre-defined static anchors, and develop a way to learn latent anchor embeddings end-to-end in the model. Lastly, we explore ensembling and output aggregation techniques—common in other ML domains—and find effective variants for our probabilistic multimodal output representation. We perform an extensive ablation on these design choices, and show that our proposed model achieves state-of-the-art performance on the Argoverse Motion Forecasting Competition [10] and the Waymo Open Dataset Motion Prediction Challenge [13].
BibTeX
@inproceedings{varadarajan2022multipath,
  author = {Balakrishnan Varadarajan and Ahmed Hefny and Avikalp Srivastava and Khaled S. Refaat and Nigamaa Nayakanti and Andre Cornman and Kan Chen and Bertrand Douillard and Chi-Pang Lam and Dragomir Anguelov and Benjamin Sapp},
  title = {{MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=n4wPPw_j4Kq},
}

Loading related content...

Figure