VLR-Driver: Large Vision-Language-Reasoning Models for Embodied Autonomous Driving

Fanjie Kong, Yitong Li, Weihuang Chen, Chen Min, Yizhe Li, Zhiqiang Gao, Haoyang Li, Zhongyu Guo, Hongbin SunICCV 2025

TL;DR

This paper introduces VLR-Driver, a novel framework that enhances autonomous driving (AD) by integrating large vision-language models (VLMs) with a Chain-of-Thought (CoT) reasoning mechanism. VLR-Driver aims to improve decision-making transparency and reliability, especially in complex and rare traffic scenarios, by generating human-like reasoning processes alongside driving actions.

What Problem Does This Paper Address?

The field of autonomous driving (AD) has seen significant progress with the advent of embodied intelligence and multi-modal large language models (LLMs). However, current AD systems, particularly those based on Vision-Language Models (VLMs), often operate as "black boxes." This lack of interpretability makes it difficult to trust their decisions, especially in intricate or ambiguous traffic situations. Furthermore, existing VLMs struggle with explaining their reasoning for rare "long-tail" events and often lack the specialized training needed for dynamic driving scenarios. This paper addresses the challenge of creating AD systems that not only make accurate decisions but also provide transparent, human-understandable reasoning, thereby increasing driver trust and safety.

Methods

VLR-Driver introduces a hierarchical Spatiotemporal Chain-of-Thought (ST-CoT) reasoning strategy. This approach breaks down the decision-making process into two levels:

  • Perception-Level CoT: This level focuses on extracting and understanding environmental dynamics. It uses multi-frame, multi-view images to identify critical objects, traffic signs, and infer the historical behavior of dynamic agents. This ensures a comprehensive understanding of the spatial layout and temporal changes in the driving environment.
  • Decision-Level CoT: This level refines and optimizes decisions based on contextual and temporal factors. It follows a structured prompt sequence: "risk point recognition," "driving intention prediction," and "driving decision selection." This hierarchical reasoning mimics human cognitive processes, leading to more interpretable and robust decisions.

The framework leverages a pre-trained LLaVA-NeXT-Video model as its VLM and CLIP for visual encoding. For training, VLR-Driver employs a dual-phase strategy:

  1. LoRA (Low-Rank Adaptation): This technique is used for efficient supervised fine-tuning, allowing adaptation with minimal memory and computational overhead.
  2. Step-GRPO (Stepwise Group Relative Policy Optimization): This reinforcement learning phase further enhances the model's reasoning and decision-making capabilities by optimizing based on human preferences and a structured, step-by-step reasoning process.

Datasets

The paper introduces the VLR-Driver Dataset, a novel benchmark designed to advance reasoning and decision-making in AD. This dataset is built upon the CARLA simulator and expanded from the Bench2Drive dataset. It includes:

  • Multi-view, multi-frame images/videos: Capturing complex traffic scenarios.
  • Detailed scene annotations: Including road conditions, weather, vehicle information, and scene descriptions.
  • Reasoning process and decision choices: Crucially, it provides human-like CoT reasoning steps and corresponding driving decisions, offering a rich foundation for training and evaluating AD systems.

The dataset comprises 20,000 sets of time-series images from over 30 complex traffic scenarios, covering various road types and weather conditions.

Key Insights and Ideas

  • Bridging the "Black Box" Gap: The core innovation is the explicit integration of CoT reasoning into VLMs for AD. This moves beyond simple prediction to provide a transparent, step-by-step explanation for driving decisions, a critical step towards trustworthy autonomous systems.
  • Spatiotemporal Reasoning is Crucial: The ST-CoT approach highlights the importance of considering both spatial relationships and temporal dynamics simultaneously for accurate and safe driving decisions, especially in dynamic environments.
  • Mimicking Human Cognition: By structuring the reasoning process into distinct stages (risk recognition, intention prediction, decision selection), the model emulates how human drivers might approach complex situations, enhancing interpretability.
  • Data for Reasoning: The creation of the VLR-Driver Dataset, with its emphasis on annotated reasoning chains, is a significant contribution, providing the necessary data to train and evaluate models on this complex task.

Key Results

  • Superior Performance: VLR-Driver significantly outperforms state-of-the-art end-to-end and VLM-based methods on the Bench2Drive benchmark.
  • Quantitative Improvements:
    • Driving Score (DS) improved by 17.5%.
  • Success Rate (SR) increased by 22.2%.
  • Mean advanced driving ability improved by 31.4%.
  • Qualitative Advantages: The model demonstrates exceptional reflective reasoning and step-by-step inference, particularly in long-tail events and rare traffic scenarios, leading to more human-like and reliable driving.

Limitations and Future Work

  • Simulation vs. Real World: The authors acknowledge the inherent differences between data collected in simulation platforms (like CARLA) and real-world driving conditions.
  • Computational Constraints: Deploying LLMs for AD on real-world vehicles with limited onboard computing resources remains a significant challenge for future exploration.
  • Further Dataset Expansion: While the VLR-Driver Dataset is substantial, continued expansion and refinement could further enhance model capabilities.

Summary updated

BibTeX
@inproceedings{kong2025vlr,
  author = {Fanjie Kong and Yitong Li and Weihuang Chen and Chen Min and Yizhe Li and Zhiqiang Gao and Haoyang Li and Zhongyu Guo and Hongbin Sun},
  title = {{VLR-Driver: Large Vision-Language-Reasoning Models for Embodied Autonomous Driving}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure