Driving Style Alignment for LLM-powered Driver Agent

Ruoxuan Yang, Xinyue Zhang, Anais Fernandez-Laaksonen, Xin Ding, Jiangtao Gong

Abstract

Recently, LLM-powered driver agents have demonstrated considerable potential in the field of autonomous driving, showcasing human-like reasoning and decision-making abilities. However, current research on aligning driver agent behaviors with human driving styles remains limited, partly due to the scarcity of high-quality natural language data from human driving behaviors. To address this research gap, we propose a multi-alignment framework designed to align driver agents with human driving styles through demonstrations and feedback. Notably, we construct a natural language dataset of human driver behaviors through naturalistic driving experiments and post-driving interviews, offering high-quality human demonstrations for LLM alignment. The framework's effectiveness is validated through simulation experiments in the CARLA urban traffic simulator and further corroborated by human evaluations. Our research offers valuable insights into designing driving agents with diverse driving styles.

Introduction

While Large Language Models (LLMs) have shown great promise in planning and reasoning for autonomous vehicles (AVs), they often lack a specific "driving style." Most current AV systems are tuned for a generic version of safety that doesn't reflect the diversity of human behavior.

The primary challenge is a data gap: most driving datasets consist of non-linguistic data (like CAN-Bus signals or video) which are difficult for LLMs to "read." To align an agent with a human style, the agent needs to understand the why behind a maneuver, not just the what. This research addresses the scarcity of high-quality natural language data regarding human driving reasoning and the lack of frameworks to efficiently synchronize LLM agents with these styles.

The Multi-Alignment Framework

The proposed Multi-Alignment Framework consists of three main components designed to bridge the gap between raw sensor data and nuanced human-like decision-making:

  1. The Driver Agent: Powered by GPT-4, this agent perceives the environment and maintains a "short-term memory" of recent situations, reasonings, and actions. It uses Chain-of-Thought (CoT) prompting to deduce the best driving action (e.g., "Given the speed of the vehicle ahead... I should match speed by gentle acceleration").
  2. The Coach Agent: This agent acts as a mentor. It reviews the Driver Agent’s recent performance against "Good" human examples and provides feedback. If the Driver Agent makes a mistake or deviates from the intended style, the Coach issues a "Bad" rating and formulates a new driving guideline.
  3. Multi-Alignment Strategy: The framework uses an ablation-tested combination of Demonstrations (D) (few-shot prompts showing how a human drives) and Feedback (F) (guidelines generated by the Coach).
The Multi-Alignment Framework architecture, illustrating the interaction between the Driver Agent, Coach Agent, and human demonstrations.

To account for the latency of GPT-4 API calls, the team implemented a "slowed-down" simulation clock in the CARLA simulator, ensuring the agent has sufficient time to process information before executing actions.

The Driving-Thinking-Dataset

A core contribution of this work is the Driving-Thinking-Dataset, a novel collection of human driving behaviors translated into natural language.

  • Data Collection: 24 drivers performed tasks in a real-world urban environment.
  • Post-Driving Interviews: Drivers participated in 1.5–2 hour "think-aloud" interviews where they watched their own driving footage and explained their decision-making process.
  • Categorization: Using the Multidimensional Driving Style Inventory (MDSI) and objective CAN-Bus data (speed, throttle %), the researchers distilled these behaviors into two primary styles: Risky and Cautious.

Experimental Evaluation and Insights

The framework was validated in the CARLA simulator and through human surveys involving 270 participants. The study compared various alignment methods—demonstrations, feedback, and the combined multi-alignment approach—across different driving styles.

Matrix of alignment methods across different driving styles (not-aligned, cautious, risky) used in the study.

Key Results

  • Safety and Distinctiveness: Cautious agents achieved the lowest collision rates under the full Multi-Alignment (D+F) condition. Conversely, Risky agents exhibited significantly higher average speeds and throttle percentages.
  • The Necessity of the "Coach": The study found that simply providing demonstrations is insufficient. The Coach Agent is vital for translating examples into high-level guidelines that the Driver Agent can follow consistently.
  • Human Perception: In double-blind tests, humans could clearly distinguish between the styles. Interestingly, participants often associated "risky" driving with higher human-likeness, even if they rated it as less "intelligent" or safe.
  • Language as an Interface: By converting driving into a "Situation-Reasoning-Action" format, the LLM leverages its pre-trained linguistic reasoning to solve physical navigation problems.

Limitations and Future Work

While promising, several hurdles remain for real-world deployment. The current system's reliance on GPT-4 introduces significant latency, requiring simulations to be slowed down. Future work will focus on improving real-time performance and exploring a wider variety of driving environments beyond the CARLA Town10 map. Additionally, researchers aim to expand the framework to include more nuanced personas, such as "Eco-friendly" or "Novice" driving styles, while addressing the high cost of current LLM API usage.

Summary updated

Abstract

Recently, LLM-powered driver agents have demonstrated considerable potential in the field of autonomous driving, showcasing human-like reasoning and decision-making abilities.However, current research on aligning driver agent behaviors with human driving styles remains limited, partly due to the scarcity of high-quality natural language data from human driving behaviors.To address this research gap, we propose a multi-alignment framework designed to align driver agents with human driving styles through demonstrations and feedback. Notably, we construct a natural language dataset of human driver behaviors through naturalistic driving experiments and post-driving interviews, offering high-quality human demonstrations for LLM alignment. The framework's effectiveness is validated through simulation experiments in the CARLA urban traffic simulator and further corroborated by human evaluations. Our research offers valuable insights into designing driving agents with diverse driving styles.The implementation of the framework and details of the dataset can be found at the link.
BibTeX
@inproceedings{yang2024driving,
  author = {Ruoxuan Yang and Xinyue Zhang and Anais Fernandez-Laaksonen and Xin Ding and Jiangtao Gong},
  title = {{Driving Style Alignment for LLM-powered Driver Agent}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=vY6ApRW69B},
}
Figure