Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving

Bob Wei, Mengye Ren, Wenyuan Zeng, Ming Liang, Bin Yang, Raquel Urtasun

TL;DR

Researchers have developed a self-driving AI called SA-NMP that uses a learned "attention mask" to focus its brainpower only on the parts of the road that actually matter for driving. By ignoring irrelevant objects (like parked cars on the opposite side of the street), the system reduces its computational load by ~80% while simultaneously becoming safer, significantly lowering collision rates compared to traditional "look-at-everything" models.

What Problem Does This Paper Address?

Most self-driving perception systems are designed to detect every single object within sensor range—every pedestrian on a far sidewalk and every parked car in a lot. While this achieves high scores on standard computer vision benchmarks (like Average Precision), it’s computationally wasteful and fundamentally misaligned with the goal of driving.

In reality, a human driver doesn't give equal weight to a car three lanes over and a car merging directly in front of them. Existing "sparse" models often use manual rules (like "only look at the road surface") to save power, but these can miss critical edge cases. The authors tackle the challenge of creating a system that learns what to prioritize based specifically on the needs of motion planning—the actual "driving" part of the stack.

Methods

The authors introduce the Sparse Attention Neural Motion Planner (SA-NMP). It builds on an end-to-end architecture that jointly handles perception, prediction, and planning.

Key technical innovations include:

  • The Attention Generator: A small U-Net sub-network that predicts a scalar "importance score" for every spatial location in a bird’s-eye-view (BEV) map.
  • Differentiable Binary Masking: To save computation, the model needs a hard "on/off" switch (a binary mask). Since standard thresholding isn't differentiable (you can't run math backwards through it to train the AI), they use the Gumbel-Softmax trick. This allows the model to learn a discrete mask while still being trainable via backpropagation.
  • Sparse Convolution Gating: The learned mask is used to gate the main backbone network. Using sparse convolution operators, the system physically skips calculations in "unattended" regions.
  • Attention-Aware Loss: They don't just use the mask for speed; they use it to weight the training process. The model is penalized more heavily for missing a "high-attention" object than a "low-attention" one, forcing the neural network to focus its limited capacity on critical actors.

Datasets

The model was evaluated on two major datasets:

  1. Drive4D: A massive real-world dataset from Uber ATG featuring over 1 million frames from 5,000 scenarios, utilizing both LiDAR and HDMaps.
  2. nuScenes: A popular large-scale public dataset. For this study, the authors used LiDAR only (omitting HDMaps due to localization inaccuracies in the public set) to test the model's robustness.

Key Insights and Ideas

  • Sparsity as a Feature, Not Just an Optimization: Usually, making a model "sparse" is a trade-off where you accept slightly lower accuracy for more speed. Here, the authors found that high sparsity (up to 95%) actually improved driving performance. By forcing the model to focus, it became less "distracted" by irrelevant data.
  • Planning-Centric Attention: Unlike prior work that used attention to improve object detection, this mask is optimized directly for the motion planning objective. The model learns to "look" where it intends to "drive."
  • Interpretability: One of the biggest "black box" problems in AI is knowing why a car made a decision. By visualizing the attention mask, engineers can see exactly which objects the car was prioritizing during a near-miss or a successful maneuver.

Key Results

  • Efficiency: The SA-NMP model used ~80% fewer FLOPs (floating-point operations) than the dense baseline while maintaining a 95% sparse attention mask.
  • Safety: On the Drive4D dataset, the learned attention model achieved a collision rate of 0.511%, significantly lower than the dense baseline (0.639%) and manual masking strategies like "Road Mask" (0.548%).
  • Accuracy: It outperformed all baselines in "Planning L2" (how closely the AI follows a human driver's path) across various computational budgets.
  • Superiority over Manual Rules: The learned mask outperformed a "Road Mask" (which only looks at the drivable surface). This is because the learned model can "peek" at sidewalks or intersections to anticipate actors about to enter the roadway.

Limitations and Future Work

The authors noted that while the model is excellent at anticipating vehicles ahead, it still struggles with certain scenarios, such as rear-end collisions where a trailing vehicle hits the self-driving car (a common issue in "open-loop" evaluations where the rest of the world doesn't react to the AI).

Additionally, the current model focuses heavily on immediate threats and sometimes ignores open road space to its side that could be used for evasive maneuvers. Future work aims to incorporate recurrent feedback, allowing the model to use information from previous time steps to better understand how a scene is evolving over time.

Summary updated

Abstract

In this paper, we propose an end-to-end self-driving network featuring a sparse attention module that learns to automatically attend to important regions of the input. The attention module specifically targets motion planning, whereas prior literature only applied attention in perception tasks. Learning an attention mask directly targeted for motion planning significantly improves the planner safety by performing more focused computation. Furthermore, visualizing the attention improves interpretability of end-to-end self-driving.
BibTeX
@inproceedings{wei2021perceive,
  author = {Bob Wei and Mengye Ren and Wenyuan Zeng and Ming Liang and Bin Yang and Raquel Urtasun},
  title = {{Perceive, Attend, and Drive: Learning Spatial Attention for Safe Self-Driving}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2021},
  url = {https://openreview.net/forum?id=zj8Jr4pRVj},
}

Loading related content...

Figure