Attention Is Not Relevance: Challenging the Causality of Attention in Autonomous Driving

Luca Lusvarghi, Javier GozalvezarXiv 2026 · arXiv

TL;DR

In autonomous driving, researchers often use "attention weights" to explain why a vehicle made a certain decision, assuming that high attention equals high relevance. This paper debunked that assumption by showing that attention is merely correlational—not causal—and frequently fails to identify the objects that actually impact a car's driving path in complex scenarios like intersections.

What Problem Does This Paper Address?

As Transformers become the backbone of autonomous driving (AD) systems, the "attention mechanism" has become a go-to tool for interpretability. If a self-driving car brakes and its attention weights are high for a nearby pedestrian, engineers often conclude the car "saw" and reacted to that pedestrian.

However, the Natural Language Processing (NLP) community has long debated whether "attention is explanation." This paper brings that skepticism to the physical world of driving. The authors tackle a critical question: Does the attention a model pays to an object actually reflect that object's causal impact on the vehicle's behavior? If attention is a poor proxy for relevance, we cannot rely on it to debug safety-critical failures or build trust in AI drivers.

Methods

To test the causality of attention, the authors established a "ground truth" for relevance using Counterfactual Analysis.

  1. The Causal Reference: For every driving scene, they iteratively removed each surrounding object and re-ran the model. If removing an object caused a significant change in the vehicle’s planned speed or path (beyond a calculated "noise floor"), that object was labeled as causally relevant.
  2. Attention-Based Methods: They evaluated three common ways to extract relevance from attention weights:
  3. Last: Using only the attention weights from the final Transformer layer.
  4. Sum: Summing raw attention weights across all heads and layers.
  5. Rollout: A more complex method that recursively multiplies attention matrices to capture how information flows through the entire network.
  6. The Model: They used PlanT 2.0, a state-of-the-art, object-level planning Transformer. PlanT 2.0 processes tokens representing the road layout, speed limits, and surrounding objects to output a planned trajectory (see Figure 1).
The PlanT 2.0 architecture uses a Transformer encoder to process various input tokens into a planned driving path.

Datasets

The study utilized over 80,000 synthetic driving scenes generated via CARLA-SUMO co-simulations.

  • Maps: CARLA’s Town01 and Town10.
  • Scenarios: A mix of single-lane and multi-lane roads, including both straightaways and complex intersections.
  • Diversity: The dataset included scenes with varying numbers of relevant objects to test how attention scales with complexity.

Key Insights and Ideas

  • Attention is a "Lazy" Proxy: In simple, single-lane scenarios, attention looks accurate because the only object present is usually the relevant one. However, this is a correlation, not a causal understanding.
  • The Complexity Trap: As driving environments get harder (e.g., multi-lane intersections), attention's accuracy plummets. It often fixates on the car directly ahead (a learned bias from training) while ignoring a "safety-critical" car crossing the intersection from the side.
  • Weak Discrimination: Even when attention correctly identifies a relevant object, the "gap" in scores between relevant and irrelevant objects is tiny compared to the actual causal impact. As shown in Figure 3, the "Relevant-to-irrelevant ratio" for actual relevance is nearly ten times higher than for any attention-based method.
Comparison of the discrimination power (Γ) between actual causal relevance and various attention methods.

Key Results

The authors found a massive performance gap between simple and complex driving tasks:

  • High Recall, Low Precision: In straight, single-lane roads, the "Sum" method achieved 98.3% recall (identifying the relevant object). However, its precision was only 27.9%, meaning it flagged many irrelevant objects as important.
  • Failure in Intersections: In scenarios with two relevant objects at an intersection, recall dropped to roughly 47-48%. Essentially, the model's attention missed half of the objects that actually mattered for safety.
  • The Tradeoff: Figure 2 illustrates a clear recall-precision tradeoff. No matter how you tune the threshold ($\beta$), attention-based methods cannot simultaneously be sensitive enough to catch all relevant objects and selective enough to ignore the noise.
Recall (a) and precision (b) curves show that attention methods struggle to accurately estimate object relevance across different thresholds.

Limitations and Future Work

The study has a few notable limitations:

  • Simulation-Only: The results are based on synthetic data from CARLA. While CARLA is high-fidelity, real-world sensor noise and occlusions might further degrade attention's reliability.
  • Single Architecture: The analysis focused on PlanT 2.0. While representative of state-of-the-art Transformers, other architectures (like those using multi-modal inputs) might behave differently.
  • Computational Cost: Counterfactual analysis is the "gold standard" for causality but is too slow for real-time use in a car.

The authors conclude that the AD community needs novel, real-time methods that can capture true causal impact rather than relying on the "mirage" of attention weights. Future work should explore how to bake causal reasoning directly into the training process of autonomous agents.

Summary updated

Abstract

Attention has become one of the most widely adopted solutions for estimating the relevance of surrounding objects and interpreting an autonomous vehicle's driving decisions. However, its ability to capture the causal impact of individual input elements on an AI model's output has long been discussed in the literature, particularly in the Natural Language Processing (NLP) domain. This work brings this discussion into the autonomous driving domain by analyzing the accuracy of attention in estimating the impact of an input object on an autonomous vehicle's driving, i.e., its relevance. To this end, we compare different attention-based relevance estimation methods against a causal reference of object relevance obtained through counterfactual analysis. We perform our analysis on more than 80,000 synthetic driving scenes using an autonomous vehicle based on PlanT 2.0, a state-of-the-art Transformer-based model for object-level planning. Results show that attention provides a correlational rather than causal indication of object relevance. Attention-based relevance estimation methods do not consistently identify all relevant objects and tend to incorrectly classify as relevant objects with little or no impact on the vehicle's driving. In particular, results show that attention can effectively identify relevant objects in simple single-lane, straight-road scenarios, but its accuracy degrades in more complex settings, such as multi-lane roads and intersections. These findings highlight the limitations of attention as a proxy for object relevance, calling for novel methods able to accurately capture the causal impact of individual objects on the vehicle's driving and distinguish it from correlational associations.
BibTeX
@misc{lusvarghi2026attention,
  author = {Luca Lusvarghi and Javier Gozalvez},
  title = {{Attention Is Not Relevance: Challenging the Causality of Attention in Autonomous Driving}},
  year = {2026},
  eprint = {2611.24051},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.24051},
}
Figure