DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks

Marco Maier, Daniel Elsner, Chadly Marouane, Meike Zehnle, Christoph FuchsIJCAI 2019 · DOI

TL;DR

This paper introduces DeepFlow, a novel end-to-end deep learning approach for automatically detecting a user's "flow state" (optimal experience, total immersion, high productivity) using physiological data from wearable sensors. By leveraging deep neural networks, DeepFlow aims to overcome the limitations of traditional self-report questionnaires, enabling real-time, unobtrusive flow detection for applications like adaptive gaming or human-in-the-loop reinforcement learning.

What Problem Does This Paper Address?

The concept of "flow" is crucial for understanding optimal user experiences, particularly in engaging activities like gaming or demanding work environments. Traditionally, assessing flow relies on self-report questionnaires, which are retrospective and require manual effort, making them unsuitable for real-time adaptation or continuous monitoring. While prior research has explored using physiological signals (like heart rate and skin conductance) to infer emotional states, including flow, these methods often rely on manual feature engineering and expensive, stationary equipment. This paper addresses the need for an automated, real-time, and accessible method for detecting flow states using readily available wearable sensors, bridging the gap between psychological theory and practical technological applications.

Methods

The core of this work is the DeepFlow model, an end-to-end deep learning architecture, specifically a Convolutional Neural Network (CNN). Unlike previous approaches that relied on handcrafted features from physiological signals, DeepFlow learns relevant features directly from raw sensor data. The network comprises multiple convolutional layers for feature extraction, followed by fully connected layers for classification.

The authors also implemented and evaluated two existing methods as benchmarks:

  • Chanel et al.'s approach: This method uses handcrafted features from physiological signals (GSR, BVP, respiration, skin temperature) and a Quadratic Discriminant Analysis (QDA) classifier.
  • Rissler et al.'s approach: This method focuses on Heart Rate Variability (HRV) features (LF, HF, LF/HF) extracted from ECG data and uses a Random Forest classifier.

For their own DeepFlow model, they explored training on three combinations of physiological signals: Blood Volume Pulse (BVP) only, Electrodermal Activity (EDA) only, and both BVP and EDA.

Datasets

The study utilized data collected from participants playing a custom version of the game Tetris. Participants were assigned to different difficulty levels designed to induce boredom, flow, or stress. Physiological data was collected using an Empatica E4 wrist-worn device, capturing signals such as BVP, EDA, and skin temperature.

  • New Benchmark/Dataset: The authors created a new dataset by collecting physiological signals synchronized with gameplay and subjective flow experience assessments (via the Game Experience Questionnaire). This dataset was used to train and evaluate their DeepFlow model and to re-evaluate the benchmark methods in their specific experimental setup. For the 3-class task (Boredom, Flow, Stress), they filtered sessions to include only those where the induced levels successfully mapped to the intended affective states, resulting in a dataset of 45 sessions.

Key Insights and Ideas

  • End-to-End Learning for Flow: The paper demonstrates that deep learning can effectively learn relevant features directly from raw physiological signals for flow detection, eliminating the need for manual feature engineering.
  • BVP vs. EDA in Flow Detection: The study found that BVP data alone performed comparably to existing methods for 2-class flow detection. Interestingly, adding EDA data improved performance for the 3-class task (Boredom, Flow, Stress) but slightly degraded it for the 2-class task. This suggests EDA might be more sensitive to the nuances of under- and over-challenged states.
  • Real-time Potential: The use of wrist-worn devices and the end-to-end deep learning approach pave the way for real-time, unobtrusive flow detection, which is a significant advancement over traditional methods.
  • Adaptability of Physiological Signals: The research highlights the potential of physiological signals to capture complex affective states beyond basic emotions, making them valuable for understanding user experience.

Key Results

  • 2-Class Task (High vs. Low Flow): DeepFlow achieved an accuracy of 67.50% using BVP and EDA combined. This is comparable to the benchmark Rissler et al. method (66.07%).
  • 3-Class Task (Boredom, Flow, Stress): DeepFlow achieved an accuracy of 49.23% using BVP and EDA combined. This is also comparable to the benchmark Chanel et al. method (48.89%).
  • The paper notes that a 50% accuracy baseline can be achieved through random chance in the 2-class task due to balanced test sets.

Limitations and Future Work

  • Standardization Dependency: The current model's reliance on standardization within sessions means it might not be directly applicable in less controlled environments where physiological baselines can vary significantly without a calibration phase.
  • Model Architecture Exploration: While the chosen CNN architecture performed well, the authors acknowledge that exploring recurrent architectures or more sophisticated deep learning models could yield further improvements.
  • Transferability: A detailed analysis of the model's transferability to other domains (e.g., office tasks) is left for future work.
  • Refinement of RL Integration: The authors suggest further research into how flow detection can be effectively integrated as a reward signal in human-in-the-loop reinforcement learning systems.
  • Data Augmentation and Tuning: Improving the dataset size and fine-tuning the model are identified as avenues for future work.

Summary updated

BibTeX
@inproceedings{maier2019deepflow,
  author = {Marco Maier and Daniel Elsner and Chadly Marouane and Meike Zehnle and Christoph Fuchs},
  title = {{DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/196},
}
Figure