DRIV-EX: Counterfactual Explanations for Driving LLMs

Amaia Cardiel, Tuan-Hung Vu, Elias Ramzi, Eric Gaussier

Abstract

Large language models (LLMs) are increasingly used as reasoning engines in autonomous driving, yet their decision-making remains opaque. We propose to study their decision process through counterfactual explanations, which identify the minimal semantic changes to a scene description required to alter a driving plan. We introduce DRIV-EX, a method that leverages gradient-based optimization on continuous embeddings to identify the input shifts required to flip the model's decision. Crucially, to avoid the incoherent text typical of unconstrained continuous optimization, DRIV-EX uses these optimized embeddings solely as a semantic guide: they are used to bias a controlled decoding process that re-generates the original scene description. This approach effectively steers the generation toward the counterfactual target while guaranteeing the linguistic fluency, domain validity, and proximity to the original input, essential for interpretability. Evaluated using the LC-LLM planner on a textual transcription of the highD dataset, DRIV-EX generates valid, fluent counterfactuals more reliably than existing baselines. It successfully exposes latent biases and provides concrete insights to improve the robustness of LLM-based driving agents.

Introduction

As Large Language Models (LLMs) transition from general-purpose assistants to reasoning engines for autonomous driving (AD), the need for interpretability becomes critical. While LLMs can provide natural language "Chain-of-Thought" (CoT) explanations for their actions, these are often post-hoc rationalizations that may not faithfully reflect the model's internal logic. A model might claim it is stopping for a pedestrian while actually being triggered by irrelevant tokens in the prompt.

Overview of the DRIV-EX framework for generating counterfactual driving scenarios.

To address this, we propose DRIV-EX, a framework for generating counterfactual explanations. Instead of asking the model "Why did you do that?", DRIV-EX identifies the minimal semantic change—such as a slight adjustment to a neighboring vehicle's speed—that would cause the model to flip its decision (e.g., from a safe lane-keep to a dangerous maneuver). By finding these decision boundaries, we can audit the true causal factors behind an LLM's driving plan.

The DRIV-EX Framework

The primary challenge in generating textual counterfactuals is the discrete nature of language. Gradient-based optimization on text often results in "gibberish" that lacks linguistic fluency or domain validity. DRIV-EX overcomes this by decoupling the search for a counterfactual from the text generation process:

  1. Continuous Optimization: The system relaxes discrete tokens into continuous embeddings and uses gradient-based optimization (via a straight-through estimator) to find the direction in embedding space that flips the model's decision.
  2. Semantic Guidance: These optimized embeddings serve as a semantic guide rather than direct output.
  3. Controlled Decoding: A fluency model re-generates the scene description, biased by the optimized embeddings. This ensures the resulting counterfactual is fluent, adheres to the driving scenario template, and remains as close as possible to the original input.

Auditing the Planner: Insights and Biases

Using the highD dataset—a collection of naturalistic vehicle trajectories—we transcribed traffic scenarios into text prompts for the LC-LLM planner. Our auditing revealed significant "unfaithful" gaps in model reasoning.

For instance, we discovered a dangerous sensitivity to lateral velocity ($v_y$) in a Llama3-based planner. DRIV-EX demonstrated that by slightly increasing the ego-vehicle's sideways speed in the prompt, the model could be tricked into making a right-lane change directly into a collision. This insight allowed for the development of a "Lateral-Agnostic" fine-tuned model, which reduced collisions in dangerous scenarios from 753 to 736 in our test subset.

Results and Performance

We compared DRIV-EX against existing baselines like DAB and PEZ across three LLM backbones: Llama3, Mistral, and Qwen2.5.

Comparative performance of DRIV-EX against state-of-the-art baselines.

As shown in the results, DRIV-EX consistently outperformed baselines in generating valid counterfactuals. On Llama3, it achieved an Aggregated Success Rate of 61.3%, significantly higher than the 49.3% achieved by the best-performing baseline. This success stems from the method's ability to maintain high fluency and similarity to the original scenario while successfully flipping the model's decision.

Conclusion

DRIV-EX represents a shift toward auditable AI in autonomous driving, moving beyond trusting what an LLM says to verifying what it actually does. While currently limited to textual descriptions and requiring white-box gradient access, future work will extend this framework to Vision-Language Models (VLMs) to handle raw sensor data, further bridging the gap between black-box reasoning and safe, interpretable autonomous systems.

Summary updated

Abstract

Large language models (LLMs) are increasingly used as reasoning engines in autonomous driving, yet their decision-making remains opaque. We propose to study their decision process through counterfactual explanations, which identify the minimal semantic changes to a scene description required to alter a driving plan. We introduce DRIV-EX, a method that leverages gradient-based optimization on continuous embeddings to identify the input shifts required to flip the model's decision. Crucially, to avoid the incoherent text typical of unconstrained continuous optimization, DRIV-EX uses these optimized embeddings solely as a semantic guide: they are used to bias a controlled decoding process that re-generates the original scene description. This approach effectively steers the generation toward the counterfactual target while guaranteeing the linguistic fluency, domain validity, and proximity to the original input, essential for interpretability. Evaluated using the LC-LLM planner on a textual transcription of the highD dataset, DRIV-EX generates valid, fluent counterfactuals more reliably than existing baselines. It successfully exposes latent biases and provides concrete insights to improve the robustness of LLM-based driving agents.
BibTeX
@inproceedings{cardiel2026driv,
  author = {Amaia Cardiel and Tuan-Hung Vu and Elias Ramzi and Eric Gaussier},
  title = {{DRIV-EX: Counterfactual Explanations for Driving LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1152},
}
Figure