Abstract
Predicting the behaviors of other agents on the road is critical for autonomous driving to ensure safety and efficiency. However, the challenging part is how to represent the social interactions between agents and output different possible trajectories with interpretability. In this paper, we introduce a neural prediction framework based on the Transformer structure to model the relationship among the interacting agents and extract the attention of the target agent on the map waypoints. Specifically, we organize the interacting agents into a graph and utilize the multi-head attention Transformer encoder to extract the relations between them. To address the multi-modality of motion prediction, we propose a multi-modal attention Transformer encoder, which modifies the multi-head attention mechanism to multi-modal attention, and each predicted trajectory is conditioned on an independent attention mode. The proposed model is validated on the Argoverse motion forecasting dataset and shows state-of-the-art prediction accuracy while maintaining a small model size and a simple training process. We also demonstrate that the multi-modal attention module can automatically identify different modes of the target agent's attention on the map, which improves the interpretability of the model.
The Challenge of Multi-modal Prediction
Predicting where a vehicle or pedestrian will go next is a fundamental challenge in autonomous driving. This difficulty stems from two primary factors: complex social interactions between agents and the inherent multi-modality of human behavior. At an intersection, a driver may turn left, turn right, or continue straight. A robust prediction model must account for all these plausible options rather than simply predicting an "average" path, which might be physically impossible or dangerous.
While many existing models rely on top-down rasterized images and heavy Convolutional Neural Networks (CNNs), these approaches are often computationally expensive and lack interpretability. This research introduces a lightweight, graph-based Transformer framework designed to be both accurate and understandable.
Methodology
The proposed framework utilizes a vectorized approach, representing the map as a series of waypoints and agents as nodes in a graph. This is significantly more efficient than image-based methods. The architecture consists of four main stages:
- Vectorized Encoding: High-resolution waypoints and agent states are encoded into features.
- Agent-Agent Encoder: A standard Transformer encoder uses multi-head attention to model how the target vehicle interacts with its neighbors.
- Multi-modal Attention Transformer: This is the core innovation. Instead of merging multiple attention heads into a single result, the authors keep them separate. Each head represents a different "mode" or intent (e.g., one head focuses on a left-turn lane, another on the straight path).
- Decoders: These separate attention features are fed into decoders to produce $K$ distinct trajectories and their associated probability scores.
To ensure the model learns diverse behaviors, a "Variety Loss" (Minimum over N loss) is used during training. Only the attention head that predicted the path closest to the actual outcome is updated, forcing the heads to specialize in different maneuvers.
Key Insights and Results
Evaluated on the Argoverse Motion Forecasting dataset, the model achieved state-of-the-art (SOTA) performance on distance-based metrics:
- minADE (Average Displacement Error): 0.8372 meters
- minFDE (Final Displacement Error): 1.2905 meters
Beyond accuracy, the research highlighted several key advantages:
- Waypoints over Lanes: High-resolution waypoints provide better results than simplified lane-level features by allowing the model to "see" the precise geometry of the road.
- Interpretability: By visualizing attention scores, the researchers demonstrated that when the model predicts a specific maneuver, the corresponding attention head is literally "looking" at the relevant waypoints on the map.
- Efficiency: The model uses only two cross-attention layers, making it lightweight enough for deployment on standard autonomous vehicle hardware.
Conclusion and Future Work
The proposed Multi-modal Transformer offers a high-performance, interpretable solution for motion prediction. While the model excels at path generation, the authors note that scoring calibration (ranking which path is most likely) remains an area for refinement. Future work will also explore scaling the model for extremely dense urban environments and incorporating additional semantic information, such as traffic light states, into the attention mechanism.
Summary updated