ADABase: A Multimodal Dataset for Cognitive Load Estimation

Maximilian P. Oppelt, Andreas Foltyn, Jessica Deuschel, Nadine R. Lang, Nina Holzer, Bjoern M. Eskofier, Seung Hee Yang2023 · DOI

TL;DR

This paper introduces ADABase, a novel multimodal dataset for estimating cognitive load in drivers, collected during simulated autonomous driving and n-back tasks. They demonstrate that combining physiological, behavioral, and performance data significantly improves cognitive load detection accuracy, paving the way for safer human-machine interaction in future vehicles.

What Problem Does This Paper Address?

As vehicles become increasingly automated, understanding and monitoring driver cognitive load is crucial for ensuring safety, especially during the transition phases where human oversight is still required. Current methods often rely on single modalities or lack comprehensive, synchronized datasets that capture the complex interplay of factors influencing cognitive load. This paper addresses the need for a rich, multimodal dataset that can support the development of robust cognitive load estimation models for autonomous driving scenarios.

Methods

The authors developed a sophisticated simulation environment for autonomous driving and utilized the well-established n-back task to induce varying levels of cognitive load. They collected data from 51 participants across multiple modalities:

  • Physiological: ECG, EDA, EMG, PPG, respiration, skin temperature.
  • Behavioral: Facial action units extracted from video.
  • Performance: Reaction time, accuracy metrics.
  • Subjective: NASA-TLX questionnaires.
  • Biomarkers: Salivary cortisol.

They employed advanced feature extraction techniques and statistical analysis (ANOVA, Friedman tests) to identify significant changes across modalities. Machine learning models, including XGBoost, SVM, and kNN, were trained and evaluated using unimodal and multimodal feature sets for both binary (low/high load) and three-level (low/medium/high load) classification tasks, as well as regression.

Datasets

The primary contribution is the ADABase dataset, a newly introduced multimodal benchmark. This dataset contains synchronized recordings from 51 subjects performing simulated driving tasks (k-drive) and n-back tasks under varying cognitive load conditions. A subset of 30 subjects' data is publicly released to facilitate further research.

Key Insights and Ideas

  • Multimodality is Key: Combining data from physiological, behavioral, and performance measures significantly boosts cognitive load estimation accuracy compared to unimodal approaches.
  • Eye Tracking is Powerful: Features derived from eye-tracking data, such as the Index of Pupillary Activity (IPA), consistently show high importance for cognitive load detection, particularly in driving scenarios.
  • Driving Simulation is Effective: The novel k-drive simulation effectively induces cognitive load and provides a realistic context for studying driver behavior.
  • Feature Importance Varies: While some features like IPA and heart rate variability are consistently important, others show task-specific relevance (e.g., skin conductance for k-drive, action units for n-back).
  • Fine-grained Classification is Challenging: While binary classification (low vs. high load) achieves high accuracy, distinguishing between multiple fine-grained levels of cognitive load presents a greater challenge, likely due to inter-subject variability.

Key Results

  • High Classification Accuracy: Multimodal models achieved an AUC of 0.91 ± 0.02 for binary classification (low vs. high cognitive load) across all tasks and levels.
  • Feature Contributions: Eye-tracking features, biosignals (ECG, EDA, PPG, EMG, RSP, SKT), and facial action units all contribute to improved classification performance.
  • Regression Performance: Regression models predicting cognitive load achieved an R² score of 0.54 ± 0.14 when using a linear combination of performance and subjective ratings as the target.
  • Statistical Significance: Numerous physiological, behavioral, and performance features showed statistically significant changes across different cognitive load levels.

Limitations and Future Work

  • Inter-Subject Variability: The authors acknowledge that inter-subject variability makes fine-grained classification (more than two levels) difficult and suggests this as an area for future work.
  • Distribution Shifts: Future research will focus on analyzing distribution shifts in machine learning models trained on one task and evaluated on another.
  • Robustness of Fusion: Investigating the robustness and stability of multimodal fusion techniques when modalities are corrupted or perturbed is a crucial next step.
  • End-to-End Models: Exploring deep neural networks for end-to-end CL prediction using raw input signals is a promising avenue.
  • Personalized Models: Incorporating personality traits into machine learning models could further enhance personalization and performance.
  • Optimizing Window Length: Further optimizing the latency and window length of extracted features could improve detection rates.

Summary updated

BibTeX
@misc{oppelt2023adabase,
  author = {Maximilian P. Oppelt and Andreas Foltyn and Jessica Deuschel and Nadine R. Lang and Nina Holzer and Bjoern M. Eskofier and Seung Hee Yang},
  title = {{ADABase: A Multimodal Dataset for Cognitive Load Estimation}},
  year = {2023},
  doi = {10.3390/s23010340},
}

Loading related content...

Figure