The Virtual Driving Instructor: Multi-Agent System Collaborating via Knowledge Graph for Scalable Driver Education

Johannes Rehm, Irina Reshodko, Stian Zimmermann Børresen, Odd Erik GundersenAAAI 2024 · DOI

TL;DR

This paper introduces a Virtual Driving Instructor (VDI) that uses a multi-agent system and a knowledge graph to provide personalized, real-time feedback to student drivers in simulators. This system aims to make driver education more scalable, standardized, and cost-effective by augmenting human instructors and offering a richer learning experience.

What Problem Does This Paper Address?

Traditional driver education, while effective, faces significant scalability and standardization challenges. A single human instructor can only effectively supervise a limited number of students simultaneously, leading to high costs and potential variability in training quality due to instructor differences and local traffic conditions. Furthermore, training for rare but critical situations, like accidents, is difficult and potentially dangerous in real-world scenarios. This paper tackles these issues by proposing an AI-powered Virtual Driving Instructor (VDI) to enhance driver education through personalized feedback and a more efficient learning environment.

Methods

The core of the VDI is a hybrid AI system that integrates several components:

  • Multi-Agent System (MAS): Over 70 agents are organized in a layered structure, handling tasks from basic reactive behaviors (e.g., monitoring traffic) to complex interpretations (e.g., analyzing overtake maneuvers). These agents interact and modify a shared knowledge graph.
  • Knowledge Graph (KG): A property graph model stores static (road layout) and dynamic (vehicle position, speed, agent reasoning) information. This KG serves as a central representation of the driving environment and the VDI's internal states, enabling comprehensive student assessment.
  • Real-time Assessment and Feedback Module (RTAFM): This module operates during the driving session, using the MAS and KG to continuously evaluate student performance and provide immediate, contextually relevant feedback. It includes a gaze estimation agent that uses a CNN to track student attention.
  • Student Model: After each session, data from the KG is used to build and update a student model based on Bayesian networks. This model tracks skill progression, identifies causal relationships between driving skills, and assesses mastery levels.
  • Driving Simulators: The system interfaces with high-fidelity driving simulators that use actual vehicles mounted on motion platforms, providing an immersive experience.
  • Teacher Station: A WebRTC-based interface allows a single instructor to monitor multiple students concurrently, receiving real-time insights and decision support from the VDI.

Datasets

Not discussed in the paper. The paper mentions that CPDs for the Bayesian network are derived from datasets generated by knowledgeable driving instructors, but no specific datasets are detailed.

Key Insights and Ideas

  • Scalability through AI Augmentation: The VDI demonstrates how AI can significantly increase the scalability of driver education, enabling a 1:3 instructor-to-student ratio, with potential for 1:5 with further refinements.
  • Standardization via Simulation: By using standardized simulated traffic scenarios, the VDI ensures a consistent learning experience regardless of location or instructor, mitigating variability inherent in traditional training.
  • Comprehensive, Real-time Feedback: The combination of a multi-agent system and a knowledge graph allows for a more nuanced and immediate assessment of student performance than previously possible.
  • The Power of a Unified System: Integrating driving simulators, a VDI, and a teacher station creates a holistic learning ecosystem that benefits both students and instructors.
  • The Importance of Contextualized Feedback: The VDI's ability to provide feedback based on specific driving situations and student attention (e.g., gaze tracking) offers more targeted guidance.

Key Results

  • Deployment: The VDI has been deployed in 5 simulators across 3 locations of a driving school in Norway since June 2021.
  • User Feedback (Survey):
    • 74% of simulator sessions were rated as good or very good.
  • Over half of students found the VDI's feedback comprehensible.
  • 55% of students partially agreed with the VDI's feedback, and 32% fully agreed.
  • User Feedback (Interviews):
    • Most interviewees found the feedback helpful and instructive.
  • Students appreciated the VDI's ability to highlight areas for improvement.
  • A preference for audio feedback over text was noted due to the focus on the road.
  • The system facilitated error recognition and correction.
  • Teacher Station Efficiency: The system allows instructors to efficiently manage multiple students, providing a detailed yet digestible snapshot of each student's performance.

Limitations and Future Work

  • Limitations:
    • Some students experienced motion sickness, limiting session duration.
  • Occasional disagreements with feedback arose, particularly when no immediate negative feedback was given for logged errors, suggesting a need to better manage student expectations.
  • The current eye-tracking system's inconsistency was a concern for mirror-checking detection.
  • Turn signal cancellation by the car's automatic systems can cause confusion.
  • The AI-generated voice for feedback was perceived as "flat" by some students.
  • Concerns about the accuracy of skill progression data in the mobile app were raised by some students.
  • Future Work:
    • Refine the system to better convey the approach of logging errors without immediate feedback to maintain student trust.
  • Improve the explanation of specific driving rules, such as turn signaling at roundabouts.
  • Address the automatic cancellation of turn signals and adjust feedback accordingly.
  • Implement a feature allowing students to revisit instructive segments of their driving sessions.
  • Integrate an interactive chatbot for enhanced two-way communication and guidance.
  • Conduct more comprehensive evaluations of the VDI's impact on on-road driving performance.

Summary updated

BibTeX
@inproceedings{rehm2024virtual,
  author = {Johannes Rehm and Irina Reshodko and Stian Zimmermann Børresen and Odd Erik Gundersen},
  title = {{The Virtual Driving Instructor: Multi-Agent System Collaborating via Knowledge Graph for Scalable Driver Education}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30315},
}
Figure