Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving

Zhengxu Tang, Xiaozhou Zhang, Guofeng Cui, Ziyu Gong, Zi Wang, Yunfei Shi, Ruifeng Deng, Chengzhi Qi, Ke Chen, Sachin Patil, Tianjun Xiao, Langechuan Liu, Pichao WangarXiv 2026 · arXiv

TL;DR

This survey reviews 171 papers to trace the evolution of Chain-of-Thought (CoT) reasoning in autonomous driving, highlighting a major shift from verbose textual explanations to "action-grounded" representations. The authors propose a taxonomy categorizing these methods into language-based, visual-spatial, latent-dynamic, and externalized reasoning. Ultimately, they argue that the future of autonomous driving intelligence relies on intermediate reasoning steps that are tightly coupled to real-time, safety-critical physical actions rather than just human-readable text.

What Problem Does This Paper Address?

Large Language Models (LLMs) and Vision-Language Models (VLMs) have popularized Chain-of-Thought (CoT) prompting—forcing a model to output its step-by-step "thinking" before delivering a final answer. Naturally, robotics and autonomous driving (AV) researchers rushed to apply this to vehicles to improve scene understanding and decision interpretability.

However, standard textual CoT is fundamentally mismatched with the demands of driving. Driving requires continuous, real-time spatiotemporal actions (trajectories), not discrete textual answers. A model can write a perfectly logical text explanation ("I see a pedestrian, I should stop") but still output a trajectory that causes a collision due to poor metric grounding. This is known as post-hoc rationalization. Furthermore, generating long text strings introduces unacceptable latency in closed-loop, real-time environments.

This survey addresses this friction by analyzing how the AV field is moving beyond text toward action-grounded reasoning representations—identifiable intermediate structures that organize driving-relevant information before a final trajectory is produced.

Figure 1: Overview of action-grounded reasoning repre- sentations in autonomous driving. Different representa- tions can connect the same scene to a driving decision. They differ in interpretability, grounding, action cou- pling, latency, and closed-loop reliability.

Methods

The authors propose a representation-centered taxonomy that classifies 130 method papers based on the form of their intermediate reasoning artifact:

  1. Language-Based CoT: Expresses reasoning via descriptive scene semantics, procedural stages (perception $\rightarrow$ prediction $\rightarrow$ planning), or reflective self-critiques.
  2. Visual-Spatial CoT: Uses future video frames, occupancy grids, or object crops as reasoning carriers to preserve metric geometry and topology.
  3. Latent-Dynamic CoT: Compresses reasoning into latent world-model rollouts, discrete tokens, or iterative trajectory optimization states (e.g., diffusion-based refinement).
  4. Externalized CoT: Delegates reasoning to external modules, such as retrieval databases (traffic rules, past driving experiences), tool calls (specialized physics planners), or multi-agent communication messages.

The paper also outlines four macro shifts currently reshaping the field:

  • The Infrastructure Shift: Moving from closed APIs (like GPT-4) to open-weight backbones (like Qwen-VL or specialized VLAs) that allow for direct fine-tuning and reinforcement learning.
  • The Cognitive Shift: Moving from simple scene description to complex future-oriented forecasting and rule-based constraints.
  • The Action Shift: Moving from generating verbal advice to outputting executable waypoints and control signals.
  • The Deployment Shift: Moving from "always-on" heavy reasoning to cost-aware adaptive reasoning that only triggers deep thinking when facing high-risk or ambiguous scenarios.

Datasets

While the paper does not introduce a new dataset, it provides a comprehensive taxonomy of 41 existing benchmarks used to evaluate driving reasoning, categorizing them by focus:

  • Reasoning & QA: Benchmarks like DriveLM, Reason2Drive, and DriveLMM-o1 that evaluate structured question-answering over driving scenes.
  • Spatial Grounding: Datasets like NuScenes-SpatialQA and DVBench that test a model's ability to localize objects and understand spatial relations.
  • Planning & Action: Standard datasets like nuScenes, nuPlan, and NAVSIM used to measure trajectory displacement errors (ADE/FDE).
  • Closed-Loop Simulation: Interactive simulators like Bench2Drive and Bench2ADVLM (built on CARLA) that evaluate how reasoning affects long-term driving scores and infraction rates.

Key Insights and Ideas

  • The Interpretability vs. Action Coupling Tension: There is a fundamental trade-off between how readable a reasoning trace is to a human and how effectively it guides a vehicle. Textual chains are highly interpretable but loosely coupled to control; latent rollouts directly influence planning but are completely opaque to human auditors.
  • Thinking Isn't Free: Deliberative reasoning (like reflection or world-model simulation) introduces massive computational latency. The authors advocate for a "System 1 + System 2" approach: using fast, reactive policies for routine lane-following, and triggering expensive reasoning loops only under high uncertainty (e.g., an un-signaled intersection).
  • Fragmentation of Evaluation: The current benchmark landscape is siloed. QA benchmarks rarely test actual driving consequences, while planning benchmarks rarely inspect whether the intermediate reasoning was correct or just a statistical shortcut.

Key Results

The survey maps out the rapid acceleration of the field, showing a massive spike in action-grounded CoT papers into 2025 and 2026. Language-based methods still dominate ($n=67$), but externalized ($n=32$), latent-dynamic ($n=18$), and visual-spatial ($n=13$) methods are growing rapidly.

The authors compile within-study performance metrics across various architectures to show the tangible benefits of action-grounded reasoning:

Representative methods per reasoning subtype and their reported performance metrics.

For instance, models utilizing reflective language reasoning (like AutoDrive-$R^2$) drastically reduce planning errors (Average L2 error down to 0.19m on nuScenes) compared to their non-reflective baselines. Similarly, adaptive switching methods (like AdaThinkDrive) manage to maintain high planning scores (90.3 PDMS on NAVSIM) while significantly cutting down on inference latency.

Limitations and Future Work

The authors note several open challenges and limitations in the current literature:

  • Lack of Causal Verification: Most current papers do not prove that their model's actions are actually caused by its reasoning.
  • Future Work - Intervention Testing: The authors call for the adoption of intervention-based faithfulness metrics. Future benchmarks should intentionally perturb the intermediate reasoning layer (e.g., deleting a detected pedestrian from the text trace or modifying a latent rollout state) and verify if the downstream trajectory changes in a predictable, causal manner.
  • Safety & Fallbacks: There is a distinct lack of formal safety guarantees or robust fallback mechanisms for when an LLM/VLM tool call fails or times out during a critical driving maneuver.

Summary updated

Abstract

Chain-of-thought (CoT) reasoning powers generative models by eliciting intermediate steps before producing an answer. In autonomous driving, the answer is a continuous action. Thus its reasoning must share the same spatiotemporal structure as the physical world. This survey studies the resulting shift from textual CoT to action-grounded reasoning. Surveying 171 papers, including 130 method papers and 41 benchmarks, datasets, surveys, and analysis papers, we propose a representation-centered taxonomy that treats the form of the intermediate state as the organizing axis. We systematize the 130 methods into four categories: language-based, visual-spatial, latent-dynamic, and externalized reasoning, further divided into 13 subtypes tied to distinct regions of interests. Our synthesis shows that the open frontier of reasoning in driving agents lies in intermediate representations that can be grounded in the real world, coupled to real-time action, and verified under safety-critical systems. Project page: https://github.com/tangzhengxu/awesome-av-cot.
BibTeX
@misc{tang2026beyond,
  author = {Zhengxu Tang and Xiaozhou Zhang and Guofeng Cui and Ziyu Gong and Zi Wang and Yunfei Shi and Ruifeng Deng and Chengzhi Qi and Ke Chen and Sachin Patil and Tianjun Xiao and Langechuan Liu and Pichao Wang},
  title = {{Beyond Textual Chain-of-Thought: A Survey on Action-Grounded Reasoning in Autonomous Driving}},
  year = {2026},
  eprint = {2609.01659},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2609.01659},
}
Figure