Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

Bob Wei, Mengye Ren, Wenyuan Zeng, Ming Liang, Bin Yang, Raquel Urtasun

Abstract

In this paper, we propose an end-to-end self-driving network featuring a sparse attention module that learns to automatically attend to important regions of the input. The attention module specifically targets motion planning, whereas prior literature only applied attention in perception tasks. Learning an attention mask directly targeted for motion planning significantly improves the planner safety by performing more focused computation. Furthermore, visualizing the attention improves interpretability of end-to-end self-driving.

Introduction

Traditional self-driving perception systems are designed to detect every object within sensor range, from distant pedestrians to parked cars in adjacent lots. While this approach yields high scores on computer vision benchmarks, it is computationally inefficient and often misaligned with the actual task of driving. In practice, a human driver prioritizes immediate threats, such as a merging vehicle, over irrelevant background objects.

To address this, researchers have developed the Sparse Attention Neural Motion Planner (SA-NMP). This system utilizes a learned "attention mask" to focus its computational resources exclusively on regions that matter for motion planning. By ignoring irrelevant data, SA-NMP reduces its computational load by approximately 80% while simultaneously improving safety and lowering collision rates.

Methodology: Learning to Attend

The SA-NMP architecture jointly handles perception, prediction, and planning. Unlike prior models that use manual rules for sparsity, SA-NMP learns what to prioritize based on the needs of the motion planner.

The SA-NMP architecture incorporates an Attention Generator and sparse convolution gating to focus processing on high-priority spatial regions.

Key technical innovations include:

  • Attention Generator: A U-Net sub-network that predicts a scalar importance score for every spatial location in a bird’s-eye-view (BEV) map.
  • Differentiable Binary Masking: To enable a hard "on/off" switch for computation, the model uses the Gumbel-Softmax trick. This allows the system to learn a discrete mask while remaining trainable via standard backpropagation.
  • Sparse Convolution Gating: The learned mask gates the main backbone network. Using sparse convolution operators, the system physically skips calculations in unattended regions, leading to significant efficiency gains.
  • Attention-Aware Loss: The model is penalized more heavily for errors in high-attention regions, forcing the network to focus its capacity on critical actors.

Experimental Results

The model was evaluated on the Drive4D dataset (containing over 1 million frames) and the nuScenes dataset. The results indicate that high sparsity (up to 95%) actually improves driving performance by reducing "distractions" from irrelevant data.

Quantitative results show that SA-NMP (blue) achieves superior safety and planning accuracy with lower computational cost compared to dense and manual masking baselines.

Key findings include:

  • Safety and Accuracy: On the Drive4D dataset, SA-NMP achieved a collision rate of 0.511%, significantly lower than the dense baseline (0.639%) and manual "Road Mask" strategies (0.548%). It also outperformed baselines in "Planning L2" accuracy across various computational budgets.
  • Efficiency: The model maintained a 95% sparse attention mask, using ~80% fewer FLOPs than the dense counterpart.
  • Interpretability: Visualizing the attention mask allows engineers to see exactly which objects the car prioritized, providing a level of transparency often missing in "black box" AI systems.

Conclusion and Future Work

SA-NMP demonstrates that a planning-centric attention mechanism can make self-driving systems both safer and more efficient. While the current model is highly effective at anticipating forward-facing threats, future iterations will aim to incorporate recurrent feedback to better manage complex temporal evolutions and interactions with trailing vehicles.

Summary updated

Abstract

In this paper, we propose an end-to-end self-driving network featuring a sparse attention module that learns to automatically attend to important regions of the input. The attention module specifically targets motion planning, whereas prior literature only applied attention in perception tasks. Learning an attention mask directly targeted for motion planning significantly improves the planner safety by performing more focused computation. Furthermore, visualizing the attention improves interpretability of end-to-end self-driving.
BibTeX
@inproceedings{wei2021perceive,
  author = {Bob Wei and Mengye Ren and Wenyuan Zeng and Ming Liang and Bin Yang and Raquel Urtasun},
  title = {{Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=zj8Jr4pRVj},
}
Figure