TL;DR
In autonomous driving, researchers often use "attention weights" to explain why a vehicle made a certain decision, assuming that high attention equals high relevance. This paper debunked that assumption by showing that attention is merely correlational—not causal—and frequently fails to identify the objects that actually impact a car's driving path in complex scenarios like intersections.
What Problem Does This Paper Address?
As Transformers become the backbone of autonomous driving (AD) systems, the "attention mechanism" has become a go-to tool for interpretability. If a self-driving car brakes and its attention weights are high for a nearby pedestrian, engineers often conclude the car "saw" and reacted to that pedestrian.
However, the Natural Language Processing (NLP) community has long debated whether "attention is explanation." This paper brings that skepticism to the physical world of driving. The authors tackle a critical question: Does the attention a model pays to an object actually reflect that object's causal impact on the vehicle's behavior? If attention is a poor proxy for relevance, we cannot rely on it to debug safety-critical failures or build trust in AI drivers.
Methods
To test the causality of attention, the authors established a "ground truth" for relevance using Counterfactual Analysis.
- The Causal Reference: For every driving scene, they iteratively removed each surrounding object and re-ran the model. If removing an object caused a significant change in the vehicle’s planned speed or path (beyond a calculated "noise floor"), that object was labeled as causally relevant.
- Attention-Based Methods: They evaluated three common ways to extract relevance from attention weights:
- Last: Using only the attention weights from the final Transformer layer.
- Sum: Summing raw attention weights across all heads and layers.
- Rollout: A more complex method that recursively multiplies attention matrices to capture how information flows through the entire network.
- The Model: They used PlanT 2.0, a state-of-the-art, object-level planning Transformer. PlanT 2.0 processes tokens representing the road layout, speed limits, and surrounding objects to output a planned trajectory (see Figure 1).
Datasets
The study utilized over 80,000 synthetic driving scenes generated via CARLA-SUMO co-simulations.
- Maps: CARLA’s Town01 and Town10.
- Scenarios: A mix of single-lane and multi-lane roads, including both straightaways and complex intersections.
- Diversity: The dataset included scenes with varying numbers of relevant objects to test how attention scales with complexity.
Key Insights and Ideas
- Attention is a "Lazy" Proxy: In simple, single-lane scenarios, attention looks accurate because the only object present is usually the relevant one. However, this is a correlation, not a causal understanding.
- The Complexity Trap: As driving environments get harder (e.g., multi-lane intersections), attention's accuracy plummets. It often fixates on the car directly ahead (a learned bias from training) while ignoring a "safety-critical" car crossing the intersection from the side.
- Weak Discrimination: Even when attention correctly identifies a relevant object, the "gap" in scores between relevant and irrelevant objects is tiny compared to the actual causal impact. As shown in Figure 3, the "Relevant-to-irrelevant ratio" for actual relevance is nearly ten times higher than for any attention-based method.
Key Results
The authors found a massive performance gap between simple and complex driving tasks:
- High Recall, Low Precision: In straight, single-lane roads, the "Sum" method achieved 98.3% recall (identifying the relevant object). However, its precision was only 27.9%, meaning it flagged many irrelevant objects as important.
- Failure in Intersections: In scenarios with two relevant objects at an intersection, recall dropped to roughly 47-48%. Essentially, the model's attention missed half of the objects that actually mattered for safety.
- The Tradeoff: Figure 2 illustrates a clear recall-precision tradeoff. No matter how you tune the threshold ($\beta$), attention-based methods cannot simultaneously be sensitive enough to catch all relevant objects and selective enough to ignore the noise.
Limitations and Future Work
The study has a few notable limitations:
- Simulation-Only: The results are based on synthetic data from CARLA. While CARLA is high-fidelity, real-world sensor noise and occlusions might further degrade attention's reliability.
- Single Architecture: The analysis focused on PlanT 2.0. While representative of state-of-the-art Transformers, other architectures (like those using multi-modal inputs) might behave differently.
- Computational Cost: Counterfactual analysis is the "gold standard" for causality but is too slow for real-time use in a car.
The authors conclude that the AD community needs novel, real-time methods that can capture true causal impact rather than relying on the "mirage" of attention weights. Future work should explore how to bake causal reasoning directly into the training process of autonomous agents.
Summary updated