Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving

Chen Lv (吕辰), Zhiyu Huang, Xiaoyu Mo, Chen Lv

TL;DR

Researchers have developed a streamlined Transformer-based framework that predicts multiple potential paths for vehicles by rethinking how "attention" works. By modifying the standard multi-head attention mechanism into a multi-modal attention layer, the model can simultaneously "look" at different parts of a map to generate distinct, interpretable trajectories for autonomous driving.

What Problem Does This Paper Address?

Predicting where a car or pedestrian will go next is one of the hardest parts of autonomous driving. The challenge is twofold:

  1. Social Interactions: Agents (cars, cyclists, pedestrians) constantly react to one another.
  2. Multi-modality: At an intersection, a driver might turn left, right, or go straight. A good model shouldn't just predict the "average" path (which might be physically impossible, like driving over a curb); it needs to predict all plausible options.

While many current models use "rasterized" images (top-down pictures of the road) and heavy Convolutional Neural Networks (CNNs), these are computationally expensive and often lack interpretability. This paper aims to create a lightweight, graph-based model that is both highly accurate and easy to understand.

Methods

The authors propose a succinct framework consisting of four main stages:

  • Vectorized Encoding: Instead of using images, they represent the map as a series of waypoints and the agents as nodes in a graph. This "vectorized" approach is much more computationally efficient.
  • Agent-Agent Encoder: A standard Transformer encoder uses multi-head attention to model how the target vehicle interacts with its neighbors.
  • The Multi-modal Attention Transformer: This is the core innovation. In a standard Transformer, multiple "attention heads" look at different things and are then merged into one result. Here, the authors keep the heads separate. Each head represents a different "mode" or intent (e.g., one head focuses on the left-turn lane, another on the straight path).
  • Decoders: These separate attention features are fed into decoders to produce $K$ different trajectories and their associated probability scores.

To ensure the model actually learns different behaviors, they use a "Variety Loss" (or Minimum over N loss) during training. Only the "winning" head—the one that predicted the path closest to what actually happened—gets its weights updated. This forces the different heads to specialize in different maneuvers.

Datasets

The model was trained and evaluated on the Argoverse Motion Forecasting dataset, a massive real-world collection containing over 324,000 driving scenarios. The task is to take 2 seconds of history and predict the next 3 seconds of movement.

Key Insights and Ideas

  • Waypoints over Lanes: The authors found that using a high-resolution list of waypoints provides better results than using simplified lane-level features. It allows the model to "see" the geometry of the road more clearly.
  • Attention as Intent: By visualizing the attention scores, the researchers proved that the model isn't just guessing. When the model predicts a left turn, the corresponding attention head is literally "looking" at the left-turn waypoints on the map. This provides a level of interpretability that many "black box" neural networks lack.
  • Efficiency is Key: Unlike many state-of-the-art models that are massive and slow, this model uses only two cross-attention layers. This makes it much easier to deploy on the limited hardware found inside an actual autonomous vehicle.

Key Results

The proposed "Multi-modal Transformer" achieved state-of-the-art (SOTA) performance on the Argoverse benchmark for distance-based metrics:

  • minADE (Average Displacement Error): 0.8372 meters.
  • minFDE (Final Displacement Error): 1.2905 meters.

These results outperformed several well-known models like LaneGCN and TNT. While its "Miss Rate" (how often it completely misses the mark) was slightly higher than the top-performing "HOME" model, the authors note that their model is significantly smaller and faster to train.

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Scoring Calibration: While the model is great at predicting the paths, its ability to rank which path is most likely (the scoring function) could be improved. They suggest that further tuning of the scoring loss weight is needed.
  • Scaling Interactions: The current model considers up to 10 surrounding agents within a 30-meter radius. In extremely dense urban environments (like a crowded city center), this might need to be expanded to capture more complex social dynamics.
  • Refining the Map: While waypoints are better than lanes, there is still room to incorporate more semantic information, such as traffic light states or temporary road construction, into the attention mechanism.

Summary updated

Abstract

Predicting the behaviors of other agents on the road is critical for autonomous driving to ensure safety and efficiency. However, the challenging part is how to represent the social interactions between agents and output different possible trajectories with interpretability. In this paper, we introduce a neural prediction framework based on the Transformer structure to model the relationship among the interacting agents and extract the attention of the target agent on the map waypoints. Specifically, we organize the interacting agents into a graph and utilize the multi-head attention Transformer encoder to extract the relations between them. To address the multi-modality of motion prediction, we propose a multi-modal attention Transformer encoder, which modifies the multi-head attention mechanism to multi-modal attention, and each predicted trajectory is conditioned on an independent attention mode. The proposed model is validated on the Argoverse motion forecasting dataset and shows state-of-the-art prediction accuracy while maintaining a small model size and a simple training process. We also demonstrate that the multi-modal attention module can automatically identify different modes of the target agent's attention on the map, which improves the interpretability of the model.
BibTeX
@inproceedings{2022multi,
  author = {Chen Lv (吕辰) and Zhiyu Huang and Xiaoyu Mo and Chen Lv},
  title = {{Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
  doi = {10.1109/icra46639.2022.9812060},
}

Loading related content...

Figure