Abstract
Fatigue is the most vital factor of road fatalities and one manifestation of fatigue during driving is drowsiness. In this paper, we propose using deep Q-learning to analyze an electroencephalogram (EEG) dataset captured during a simulated endurance driving test. By measuring the correlation between drowsiness and driving performance, this experiment represents an important brain-computer interface (BCI) paradigm especially from an application perspective. We adapt the terminologies in the driving test to fit the reinforcement learning framework, thus formulate the drowsiness estimation problem as an optimization of a Q-learning task. By referring to the latest deep Q-Learning technologies and attending to the characteristics of EEG data, we tailor a deep Q-network for action proposition that can indirectly estimate drowsiness. Our results show that the trained model can trace the variations of mind state in a satisfactory way against the testing EEG data, which demonstrates the feasibility and practicability of this new computation paradigm. We also show that our method outperforms the supervised learning counterpart and is superior for real applications. To the best of our knowledge, we are the first to introduce the deep reinforcement learning method to this BCI scenario, and our method can be potentially generalized to other BCI cases.
Introduction
Drowsiness is a leading cause of road fatalities, and Electroencephalogram (EEG) data—which measures brain activity—is considered the "gold standard" for detecting it. However, current machine learning approaches face two major hurdles. First is the Label Gap: traditional Supervised Learning (SL) requires every snippet of EEG data to be paired with a "ground-truth" label like a reaction time. In a 90-minute drive, long stretches of data lack labels and are typically discarded. Second is the Generalization Gap: models trained on isolated trials often struggle with continuous, real-time data streams where signal distributions change over time. This paper proposes a Reinforcement Learning (RL) framework that treats drowsiness estimation as a sequential decision-making task, allowing for a global tracing of mental state changes.
Methodology
The core innovation of this work is framing drowsiness estimation as a "game" played by a Deep Q-Network (DQN). Instead of simple regression, the problem is modeled as an agent interacting with an EEG stream environment.
The Reinforcement Learning Framework
The agent manages an internal "Response Time (RT) tracer." Its goal is to adjust this tracer to match the driver's actual reaction time whenever a lane-deviation event occurs. If the traced RT is close to the measured RT, the agent receives a reward; otherwise, it receives a penalty. During intervals between events, the reward is zero, but the agent learns to maintain its state based on the continuous EEG input.
Architecture
The model utilizes a Recurrent Convolutional Neural Network (RCNN). A CNN extracts spatial features from the EEG electrodes, while a 2D Long Short-Term Memory (LSTM) network captures the temporal dynamics of the brain's state. To improve stability, a Dueling DQN architecture is employed, which separately estimates the value of a state and the advantage of specific actions (e.g., increasing the predicted RT).
Experimental Setup
The study utilized an EEG dataset from 37 subjects performing a simulated endurance driving task. The setup involved a real car mounted on a 6-degree-of-freedom platform facing a 4-lane highway simulation. Drivers were tasked with cruising in a lane while random "turbulence" caused the car to drift. The Response Time (RT)—the time taken to notice and correct the drift—served as the proxy for drowsiness.
EEG data was recorded using 32 channels at 500Hz, later downsampled to 128Hz and bandpass filtered (0.5–50 Hz). The RL agent processes the entire session as a sequence, utilizing the "unlabeled" data between driving events to learn the transitions between alertness and fatigue.
Key Results and Discussion
The RL approach proved significantly more robust than traditional Supervised Learning, particularly in cross-subject scenarios.
- Tracing vs. Point-Estimation: Unlike SL, which seeks a single correct number for a data snippet, the RL agent learns a policy to update its estimate based on brain activity trends.
- Correlation and Robustness: In cross-subject tests, the RL model achieved correlation coefficients 3x to 6x higher than SL counterparts. For example, in one subject (S50), the RL model achieved a 0.45 correlation while the SL model failed to track the trend (0.07 correlation).
- Transition Weight (β): A weight of 0.75 (favoring historical state over new evidence) was found to be optimal, confirming that mental states change procedurally rather than instantaneously.
Conclusion and Future Work
This study demonstrates that Deep Reinforcement Learning is a viable and superior paradigm for tracing mental states in BCI applications. While the current model faces challenges like "jittery" predictions and quantization errors due to a discrete action space, future work will explore Actor-Critic methods for continuous action spaces and model-based RL to better predict state transitions. This framework represents a significant step toward more practical and efficient real-time drowsiness detection systems.
Summary updated