MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction

Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi-Pang Lam, Dragomir Anguelov, Benjamin Sapp

Abstract

Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model that achieves state-of-the-art performance on popular benchmarks. MultiPath++ improves the MultiPath architecture by revisiting many design choices. The first key design difference is a departure from dense image-based encoding of the input world state in favor of a sparse encoding of heterogeneous scene elements: MultiPath++ consumes compact and efficient polylines to describe road features, and raw agent state information directly (e.g., position, velocity, acceleration). We propose a context-aware fusion of these elements and develop a reusable multi-context gating fusion component. Second, we reconsider the choice of pre-defined, static anchors, and develop a way to learn latent anchor embeddings end-to-end in the model. Lastly, we explore ensembling and output aggregation techniques -- common in other ML domains -- and find effective variants for our probabilistic multimodal output representation. We perform an extensive ablation on these design choices, and show that our proposed model achieves state-of-the-art performance on the Argoverse Motion Forecasting Competition and the Waymo Open Dataset Motion Prediction Challenge.

Introduction

Predicting the future trajectories of road users—such as vehicles, cyclists, and pedestrians—is a critical yet daunting task in autonomous driving. The environment is inherently stochastic, where a driver's intent can shift in an instant, and heterogeneous, requiring the seamless fusion of map data, traffic signals, and the movement history of numerous agents.

Traditional state-of-the-art models, including the original MultiPath, often relied on "rasterization," which converts the scene into a top-down image processed by standard Convolutional Neural Networks (CNNs). However, rasterization is computationally inefficient due to sparse data, loses precision through grid resolution, and struggles with long-range interactions. MultiPath++ addresses these limitations by moving toward sparse representations and efficient information fusion.

Methodology

MultiPath++ introduces a streamlined architecture designed for efficiency and precision. The model shifts from dense image-based encodings to sparse "polyline" encodings for road features and raw state sequences for agents. This approach preserves geometric detail while reducing computational overhead.

High-level architecture of MultiPath++ showing the flow from sparse inputs to probabilistic trajectory outputs.

The architecture consists of three main stages:

  1. Sparse Encoding: Road features are represented as polylines, and agent histories are processed as raw state sequences (position, velocity, etc.).
  2. Multi-Context Gating (MCG): A novel fusion engine that aggregates scene information without the quadratic cost of all-to-all attention.
  3. Predictor with Learned Anchors: Instead of static, pre-defined trajectory templates, the model learns latent anchor embeddings end-to-end, allowing it to adapt to diverse driving scenarios.

Multi-Context Gating (MCG)

A core innovation of MultiPath++ is the Multi-Context Gating (MCG) mechanism. In complex scenes, standard cross-attention mechanisms can become computationally prohibitive as the number of elements increases. MCG provides a more efficient alternative by creating a global "context vector" that summarizes the scene. This vector is then used to gate or filter the features of individual elements.

Comparison between standard cross-attention and the more efficient Multi-Context Gating (MCG) mechanism.

By using this summary-based approach, the model achieves the benefits of contextual awareness—understanding how a lane's relevance changes based on nearby agent behavior—at a significantly lower computational cost than all-to-all attention.

Evaluation and Results

The model was rigorously tested on the Waymo Open Motion Dataset (WOMD) and the Argoverse Motion Forecasting Dataset. MultiPath++ demonstrated dominant performance:

  • Waymo Leaderboard: Ranked 1st overall at publication, achieving a Mean Average Precision (mAP) of 0.409 and a Minimum Average Distance Error (minADE) of 0.556m.
  • Argoverse Leaderboard: Ranked 4th overall, showcasing strong generalization across different map formats and sensor configurations.

Ablation studies confirmed that the combination of stacked MCG blocks and learned latent anchors provided the most significant improvements in accuracy and miss rate. Furthermore, the use of an EM-based ensembling technique ensured that the final output remained a diverse and representative set of possible futures, avoiding the "averaging" trap that can lead to physically impossible paths.

Conclusion and Future Work

MultiPath++ proves that sparse, object-centric representations are superior to dense rasters for motion prediction. While the model sets a new benchmark for efficiency and accuracy, challenges remain. Future work includes improving kinematic feasibility—ensuring predicted paths are always physically achievable—and exploring joint prediction models that explicitly account for the interactive futures of all agents in a scene.

Summary updated

Abstract

Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model that achieves state-of-the-art performance on popular benchmarks. MultiPath++ improves the MultiPath architecture by revisiting many design choices. The first key design difference is a departure from dense image-based encoding of the input world state in favor of a sparse encoding of heterogeneous scene elements: MultiPath++ consumes compact and efficient polylines to describe road features, and raw agent state information directly (e.g., position, velocity, acceleration). We propose a context-aware fusion of these elements and develop a reusable multi-context gating fusion component. Second, we reconsider the choice of pre-defined, static anchors, and develop a way to learn latent anchor embeddings end-to-end in the model. Lastly, we explore ensembling and output aggregation techniques -- common in other ML domains -- and find effective variants for our probabilistic multimodal output representation. We perform an extensive ablation on these design choices, and show that our proposed model achieves state-of-the-art performance on the Argoverse Motion Forecasting Competition and the Waymo Open Dataset Motion Prediction Challenge.
BibTeX
@inproceedings{varadarajan2022multipath,
  author = {Balakrishnan Varadarajan and Ahmed Hefny and Avikalp Srivastava and Khaled S. Refaat and Nigamaa Nayakanti and Andre Cornman and Kan Chen and Bertrand Douillard and Chi-Pang Lam and Dragomir Anguelov and Benjamin Sapp},
  title = {{MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  url = {https://openreview.net/forum?id=n4wPPw_j4Kq},
}
Figure