Cockpit-Llama: Driver Intent Prediction in Intelligent Cockpit via Large Language Model

Yi Chen, Chengzhe Li, Qirui Yuan, Jinyu Li, Yuze Fan, Xiaojun Ge, Yun Li, Fei Gao, Rui ZhaoSensors 2024 · DOI

TL;DR

This paper introduces Cockpit-Llama, a novel approach using Large Language Models (LLMs) to predict driver intent within intelligent cockpits. By integrating diverse contextual data and employing efficient fine-tuning techniques, Cockpit-Llama significantly outperforms existing methods, paving the way for more proactive and intuitive human-machine interactions in vehicles.

What Problem Does This Paper Address?

Modern intelligent cockpits are moving beyond simple command execution towards proactive assistance. To achieve this, accurately predicting a driver's intent is crucial. However, current methods often struggle with the complexity and interconnectedness of driver actions, environmental states, and internal cockpit conditions. Existing approaches are often limited to single-task predictions, lack transparency (acting as "black boxes"), and fail to incorporate the rich social and contextual understanding that humans possess. This gap hinders the development of truly intelligent and responsive cockpit experiences.

Methods

The core of this work is Cockpit-Llama, a system built upon the Llama3-8b-Instruct LLM. Its architecture comprises three main components:

  1. Modal Text Data Encoding: All relevant data (historical interactions, cockpit states, environmental factors) is first converted into a unified text format. This includes a novel instruction text that guides the LLM.
  2. Lightweight Fine-Tuning (LoRA): To adapt the powerful Llama3 backbone to the specific task of cockpit intent prediction without incurring prohibitive computational costs, the authors employ the Low-Rank Adaptation (LoRA) method. LoRA efficiently fine-tunes the model by adding small, low-rank matrices to the existing weight matrices, keeping the original LLM parameters frozen. This significantly reduces training costs while maintaining high performance.
  3. Driver Behavior Intent Decoding: The LLM's output (predicted tokens) is then decoded into human-readable language, providing interpretable predictions of the driver's future actions.

The system leverages a multi-attribute cockpit dataset, which includes driver emotional states, driving scenarios, vehicle motion, body states, multimedia usage, and external environmental conditions.

Datasets

The authors introduce a novel multi-attribute cockpit dataset for training and evaluating Cockpit-Llama. This dataset comprises approximately 60,000 sequences of multi-attribute states and historical driver interactions. It was collected over a month using five experimental vehicles and fifteen recruited individuals, capturing natural driving experiments. The dataset includes:

  • Driver emotional states: (e.g., anger, happiness, sadness)
  • Driving activity scenarios: (e.g., daily, commute, far driving)
  • Vehicle motion states: (e.g., speed, steering angle)
  • Body states: (e.g., seat position, window status)
  • Multimedia states: (e.g., music, radio, navigation)
  • External environment: (e.g., temperature, light)

This dataset is crucial for fine-tuning Cockpit-Llama and enabling it to understand the complex interplay of factors influencing driver intent.

Key Insights and Ideas

  • LLMs as a Bridge for Contextual Understanding: The paper highlights how LLMs, with their pre-training on vast interdisciplinary data, inherently possess a "common-sense" understanding of human behavior that traditional models lack. This is key to bridging the "social intelligence gap" in predictive systems.
  • Multi-Attribute Data is Crucial: The success of Cockpit-Llama hinges on its ability to process a rich tapestry of data, not just isolated sensor readings. Integrating emotional states, driving scenarios, and environmental factors provides the necessary context for accurate intent prediction.
  • LoRA for Efficient LLM Adaptation: The use of LoRA demonstrates a practical and effective way to fine-tune large LLMs for specialized tasks like driver intent prediction, making advanced AI accessible without massive computational resources. This is a significant contribution for applying LLMs in resource-constrained environments like vehicles.
  • Interpretability through Natural Language Output: Decoding the LLM's predictions into natural language makes the system's reasoning process more transparent and understandable to humans, a critical factor for trust and adoption in safety-critical applications.

Key Results

Cockpit-Llama demonstrated superior performance across multiple evaluation metrics compared to advanced LLMs like ChatGPT-4 and ChatGLM-3:

  • BLEU-4: 71.32 (a 92.34% improvement over ChatGPT-4)
  • ROUGE-1: 80.01 (a 183.61% improvement over ChatGPT-4)
  • ROUGE-2: 76.89 (a 95.54% improvement over ChatGPT-4)
  • ROUGE-L: 81.42 (a 201.27% improvement over ChatGPT-4)

These results indicate a significantly higher similarity and overlap between Cockpit-Llama's predicted intents and the ground truth, showcasing its accuracy and effectiveness. The paper also showed that LoRA fine-tuning significantly outperformed in-context learning and freeze fine-tuning methods.

Limitations and Future Work

The authors acknowledge several limitations:

  • Dataset Size: The current dataset, while novel and comprehensive, has "only about 60,000 samples." This is considered limited for fully capturing the nuances of driver behavior.
  • Inference Time: The inference time is noted as "relatively long" due to parameter size, which may not meet real-time requirements for commercial driving applications.
  • Complexity of Driver Behavior: The model may not yet fully capture the intricate complexity of driver behavior and all influencing factors.

Future work directions include:

  • Constructing Larger-Scale Datasets: To provide more prior knowledge for the model.
  • Optimizing Inference Time: Exploring methods like using LLMs to guide smaller models or other compression techniques.
  • Integrating Real-time Data and Additional Contextual Variables: To further enhance predictive accuracy.

Summary updated

Abstract

The cockpit is evolving from passive, reactive interaction toward proactive, cognitive interaction, making precise predictions of driver intent a key factor in enhancing proactive interaction experiences. This paper introduces Cockpit-Llama, a novel language model specifically designed for predicting driver behavior intent. Cockpit-Llama predicts driver intent based on the relationship between current driver actions, historical interactions, and the states of the driver and cockpit environment, thereby supporting further proactive interaction decisions. To improve the accuracy and rationality of Cockpit-Llama's predictions, we construct a new multi-attribute cockpit dataset that includes extensive historical interactions and multi-attribute states, such as driver emotional states, driving activity scenarios, vehicle motion states, body states and external environment, to support the fine-tuning of Cockpit-Llama. During fine-tuning, we adopt the Low-Rank Adaptation (LoRA) method to efficiently optimize the parameters of the Llama3-8b-Instruct model, significantly reducing training costs. Extensive experiments on the multi-attribute cockpit dataset demonstrate that Cockpit-Llama's prediction performance surpasses other advanced methods, achieving BLEU-4, ROUGE-1, ROUGE-2, and ROUGE-L scores of 71.32, 80.01, 76.89, and 81.42, respectively, with relative improvements of 92.34%, 183.61%, 95.54%, and 201.27% compared to ChatGPT-4. This significantly enhances the reasoning and interpretative capabilities of intelligent cockpits.
BibTeX
@inproceedings{chen2024cockpit,
  author = {Yi Chen and Chengzhe Li and Qirui Yuan and Jinyu Li and Yuze Fan and Xiaojun Ge and Yun Li and Fei Gao and Rui Zhao},
  title = {{Cockpit-Llama: Driver Intent Prediction in Intelligent Cockpit via Large Language Model}},
  booktitle = {Sensors},
  year = {2024},
  doi = {10.3390/s25010064},
}
Figure