Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions

Shiyu Fang, Jiaqi Liu, Chengkai Xu, Chen Lv, Peng Hang, Jian Sun

Introduction

Autonomous vehicles (AVs) have achieved significant proficiency in adhering to the formal rules of the road. However, they continue to struggle with the "social" nuances of driving—the subtle negotiations required during lane merges, intersection crossings, and interactions with human drivers. While Large Language Models (LLMs) offer a promising path toward human-like reasoning and intent understanding, their high computational latency makes them unsuitable for the split-second decision-making required in dynamic traffic.

This paper introduces the "Actor-Reasoner" framework, a dual-system architecture that bridges the gap between high-level cognitive reasoning and real-time execution. By decoupling deliberate reasoning from intuitive action, the framework enables AVs to interact safely, interpretably, and efficiently with human-driven vehicles (HVs).

The Challenge: Balancing Reasoning and Real-Time Response

Human drivers utilize a combination of subtle cues and explicit communication to navigate complex scenarios. Current AV systems often default to extreme caution, leading to "deadlocks," or behave unpredictably because they cannot distinguish between different driving styles (e.g., aggressive vs. conservative).

While LLMs excel at Chain-of-Thought (CoT) reasoning and understanding human instructions, the several-second delay inherent in LLM inference poses a safety hazard at high speeds. The Actor-Reasoner framework addresses this conflict by mimicking the human cognitive model of System 1 (fast, intuitive) and System 2 (slow, deliberate) thinking.

The Actor-Reasoner Architecture

The proposed framework operates through two parallel modules designed to handle different temporal demands:

1. The Reasoner (System 2)

Powered by a localized Llama3-7B model, the Reasoner handles high-level strategy. It processes environmental context and human voice-to-text instructions to infer the driving styles of nearby vehicles. Based on this analysis, it determines the appropriate communication strategy for an external Human-Machine Interface (eHMI), such as displaying "I will go first" to resolve ambiguity. Because it operates in a parallel thread, its "slow" thinking does not stall the vehicle's immediate responses.

2. The Actor (System 1)

The Actor is a lightweight, memory-retrieval module responsible for millisecond-level control. Rather than generating actions from scratch, it utilizes a Two-layer Memory Retrieval (TMemR) process:

  • Layer 1: Filters the Interaction Memory Database based on numerical data (position, velocity).
  • Layer 2: Refines the search using vectorized "experience descriptions"—textual reasoning provided by the LLM—to identify the most appropriate historical action.

Memory Partitioning

To optimize retrieval, the database is partitioned by driving styles (Aggressive, Normal, Conservative). When the Reasoner identifies a nearby driver as "Aggressive," the Actor restricts its search to the corresponding memory block. This partitioning prevents the AV from applying conservative logic to aggressive opponents, reducing the likelihood of deadlocks and increasing retrieval speed by an average of 12%.

Experimental Validation

The researchers evaluated the framework using both high-fidelity simulations and real-world field tests at Tongji Small Town (TJST) in Shanghai. The training environment utilized non-cooperative Bayesian games and Inverse Reinforcement Learning to simulate a diverse range of human driving behaviors.

Key Results

  • Safety: In multi-vehicle intersection scenarios, the Actor-Reasoner framework achieved a dangerous interaction rate of only 1%. This significantly outperformed traditional Reinforcement Learning methods like PPO (7%) and other LLM-based approaches like Dilu (11%).
  • Efficiency: The system maintained an average travel velocity of 4.0 m/s, the highest among all tested safe methods, demonstrating that safety does not have to come at the cost of traffic flow.
  • Computational Feasibility: The memory partitioning module achieved peak retrieval speed improvements of up to 19%, making real-time LLM-driven interaction a practical reality.

Conclusion and Future Directions

The Actor-Reasoner framework demonstrates that LLMs can be effectively integrated into autonomous driving systems by separating high-level reasoning from low-level execution. By treating driving as a social negotiation rather than a purely geometric problem, the system achieves superior safety and efficiency.

Future work will focus on optimizing eHMI displays to ensure human drivers can process AV intentions even more rapidly. Additionally, the authors plan to expand the framework toward "collaborative driving," where multiple autonomous agents negotiate collectively to optimize intersection throughput and model the gradual development of trust between humans and AI drivers.

Summary updated

Abstract

Autonomous Vehicles (AVs) have entered the stage of commercialization, yet their performance in interactive scenarios remains unsatisfactory due to challenges such as decision interpretability, human driver (HV) heterogeneity, and scenario diversity. Recent advances in Large Language Models (LLMs) provide a promising avenue to enhance AV interaction capabilities, but their high computational demand hinders practical deployment. To address these challenges, this paper introduces a parallel Actor–Reasoner framework designed to enable explicit and real-time bidirectional AV-HV interactions. First, the Reasoner employs a localized LLM with CoT reasoning and human instructions to progressively infer HV intent, style, AV action, and eHMI displays during the training stage. During testing, it continues to infer he above information except AV action. The Actor, in turn, is constructed as an interaction memory through the Reasoner’s interactions with heterogeneous simulated HVs across diverse scenarios, where the memory partition and two-layer retrieval modules are employed in the construction process. During testing, the Actor is used to retrieve feasible actions for the AV. Ablation studies across multiple scenarios demonstrate that the proposed modules improve interaction success rates by an average of 15% and 12%, respectively. Moreover, comparison studies in multi-vehicle scenarios further show that the proposed Actor–Reasoner framework achieves superior safety while simultaneously improving efficiency. Finally, with the integration of external Human–Machine Interface (eHMI) information derived from the Reasoner’s reasoning and feasible actions retrieved from the Actor, the framework is validated in real-world field interactions. Our code is available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/FanGShiYuu/Actor-Reasoner</uri>
BibTeX
@article{fang2025interact,
  author = {Shiyu Fang and Jiaqi Liu and Chengkai Xu and Chen Lv and Peng Hang and Jian Sun},
  title = {{Interact, Instruct to Improve: A LLM-Driven Parallel Actor-Reasoner Framework for Enhancing Autonomous Vehicle Interactions}},
  journal = {IEEE Transactions on Intelligent Transportation Systems},
  year = {2025},
  doi = {10.1109/tits.2025.3633264},
}
Figure