Passenger hazard perception based on EEG signals for highly automated driving vehicles

Ashton Yu Xuan Tan, Yingkai Yang, Xiaofei Zhang, Bowen Li, Xiaorong Gao, Sifa Zheng, Jianqiang Wang, Xinyu Gu, Jun Li, Zaili Yang, Yuxin Zhang, Tania Stathaki

TL;DR

This paper introduces a novel approach to enhance autonomous vehicle (AV) safety by decoding passenger brain activity (EEG) to predict and identify hazardous driving scenarios. They developed a Passenger Cognitive Model (PCM) and a Passenger EEG Decoding Strategy (PEDS) utilizing a new Convolutional Recurrent Neural Network (CRNN) that achieves 85% accuracy in identifying dangerous situations, highlighting the potential of pre-event EEG data for proactive safety measures.

What Problem Does This Paper Address?

The increasing adoption of highly automated driving vehicles (AVs) has been marred by safety concerns, particularly "Safety of The Intended Functionality" (SOTIF) accidents. These incidents often stem from AVs failing to handle unforeseen or unknown scenarios, a limitation of current AI algorithms that rely on probabilistic statistics and struggle with the complexities of real-world driving. While drivers are the primary focus of safety research, passengers also play a crucial role, especially as AVs become more prevalent. Passengers can act as "sensors" and "decision-makers," and their risk perception and cognitive states could be integrated into AV systems to improve safety. However, research on passenger hazard perception in AVs, particularly in dynamic and unpredictable situations, is limited. This paper aims to bridge this gap by exploring the neural mechanisms behind passenger hazard perception and developing a system to leverage this information for enhanced AV safety.

Methods

The core of this research lies in developing a Passenger Cognitive Model (PCM) and a Passenger EEG Decoding Strategy (PEDS).

  • Passenger Cognitive Model (PCM): This model is built upon analyzing EEG topographic maps and understanding the functionalities of different brain regions. It categorizes passenger cognitive states into "Calm" and "Nervous" and further breaks them down into specific phases like "Navigating with no surrounding targets," "Approaching Target," "Overtaking Target," and "Leaving Target." This model aims to elucidate the intricate neural mechanisms underlying passenger interactions with AVs across diverse traffic conditions, distinguishing between "risk prediction" and "danger identification."
  • Passenger EEG Decoding Strategy (PEDS): PEDS utilizes the PCM to define two key classification tasks:
  • Risk Prediction: This task aims to detect potential hazards before an event occurs, using EEG data from the time window of -2s to -0.5s before the event. It classifies scenarios as "Risky" (low-risk and high-risk) or "Not Risky" (safe).
  • Danger Identification: This task focuses on identifying actual hazardous behaviors during an event, using EEG data from -0.2s to 1s. It classifies scenarios as "Dangerous" (high-risk) or "Not Dangerous" (safe and low-risk).

    For these tasks, the researchers explored several machine learning approaches:

  • CSP + SVM: A common spatial pattern (CSP) algorithm combined with a support vector machine (SVM) for binary classification.

  • XGB-DIM: An eXtreme Gradient Boosting Discriminative Information Mining model.
  • Stacking Algorithms: Two stacking approaches were investigated:
  • Stacking Algorithm 1 (Weighted Sums): Combines CSP+SVM for risk prediction with XGB-DIM for danger identification, using a weighted sum of their outputs.
  • Stacking Algorithm 2 (Meta-Learner): Uses CSP+SVM for risk prediction and integrates advanced CNN models (EEGNet, ShallowConvNet, DeepConvNet) as base models for danger identification, with a logistic regression meta-learner.
  • Novel Convolutional Recurrent Neural Network (CRNN): This is the paper's main novel contribution. It integrates two independent CNN encoders for spatial and temporal feature extraction from EEG data across different time windows, followed by a Gated Recurrent Unit (GRU)-based RNN decoder for temporal dependency modeling. This CRNN model is designed for end-to-end optimization.

Datasets

The study utilized a newly created EEG dataset comprising 14 distinct traffic scenarios collected within a driving simulator. These scenarios were designed to represent various autonomous driving situations, including pedestrian crossings, vehicles cutting in or out, and emergency braking. The dataset was collected from 15 participants (12 males, 3 females, mean age 27.8 years), with data from 11 participants deemed valid for analysis. The experiment involved participants sitting in a driving simulator and responding to auditory cues. The data segments were labeled as "safe," "low-risk," or "high-risk" based on predefined definitions related to the presence of traffic participants and dangerous behaviors.

Key Insights and Ideas

  • Pre-event EEG is predictive: The study strongly suggests that EEG signals before a hazardous event can predict its occurrence, offering a crucial window for proactive intervention. This "risk prediction" capability is a significant advancement over reactive systems.
  • Passenger cognition is complex and dynamic: The PCM effectively illustrates how passenger mental states transition from "Calm" to "Nervous" in response to approaching and overtaking targets, highlighting distinct neural patterns associated with risk assessment and danger identification.
  • P300 component is a key indicator: The P300 component, a well-known event-related potential, was observed in the EEG topographic maps during highly dangerous scenarios, indicating heightened cognitive engagement and visual stimulus processing.
  • Brain regions show specific roles: The research highlights the involvement of frontal lobes for predictive functions and eye movement control, and occipital/parietal lobes for visual processing, aligning with the PCM's insights.
  • CRNN excels at spatio-temporal EEG analysis: The novel CRNN architecture effectively captures both spatial and temporal patterns in EEG data, outperforming other models in distinguishing dangerous scenarios.

Key Results

  • The proposed CRNN model achieved a balance accuracy of 85.0% ±3.18% for the combined risk prediction and danger identification tasks.
  • The CRNN model demonstrated superior performance across subjects, with a True Positive Rate (TPR) of 90.3%, significantly higher than other state-of-the-art models.
  • The CRNN also exhibited the lowest average False Positive Rate (FPR) of 20.3% ±2.53%, indicating fewer unnecessary interventions.
  • Stacking algorithms also showed significant improvements, with one approach reducing the FPR for danger identification from 47.6% to 7.20% while maintaining TPR.
  • The peak latency analysis identified distinct time windows for risk prediction (-2s to -0.5s) and danger identification (-0.2s to 1s), crucial for optimizing classification.

Limitations and Future Work

  • Intersubject Variability: The authors acknowledge that EEG responses vary significantly among individuals. While their approach uses participant-specific data and the CRNN shows robustness, this remains a challenge.
  • Fatigue and Vigilance: Prolonged experimental sessions can lead to fatigue and reduced vigilance, potentially affecting EEG responses. The study suggests participants take breaks, but controlling fatigue remains difficult.
  • Real-time Implementation: The current study was conducted in a laboratory setting. Future work will focus on achieving real-time data acquisition and analysis for practical AV integration.
  • Multimodal Integration: The authors recognize that relying solely on EEG is insufficient. Future research will explore integrating other signal types, such as electromyography (EMG) and functional near-infrared spectroscopy (fNIRS), to create a more robust decision-making system.
  • Advanced Classification: Future development of PEDS aims to extract more nuanced information, such as distinguishing between vehicle types or identifying the orientation of hazards.
  • Reinforcement Learning: The integration of PEDS with reinforcement learning algorithms is proposed to enable AVs to learn and optimize their decision-making behavior in complex environments.

Summary updated

Abstract

Enhancing the safety of autonomous vehicles is crucial, especially given recent accidents involving automated systems. As passengers in these vehicles, humans' sensory perception and decision-making can be integrated with autonomous systems to improve safety. This study explores neural mechanisms in passenger-vehicle interactions, leading to the development of a Passenger Cognitive Model (PCM) and the Passenger EEG Decoding Strategy (PEDS). Central to PEDS is a novel Convolutional Recurrent Neural Network (CRNN) that captures spatial and temporal EEG data patterns. The CRNN, combined with stacking algorithms, achieves an accuracy of $85.0\% \pm 3.18\%$. Our findings highlight the predictive power of pre-event EEG data, enhancing the detection of hazardous scenarios and offering a network-driven framework for safer autonomous vehicles.
BibTeX
@misc{tan2024passenger,
  author = {Ashton Yu Xuan Tan and Yingkai Yang and Xiaofei Zhang and Bowen Li and Xiaorong Gao and Sifa Zheng and Jianqiang Wang and Xinyu Gu and Jun Li and Zaili Yang and Yuxin Zhang and Tania Stathaki},
  title = {{Passenger hazard perception based on EEG signals for highly automated driving vehicles}},
  year = {2024},
  eprint = {2408.16315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.16315},
}
Figure