Toward Dynamically Adaptive Simulation: Multimodal Classification of User Expertise Using Wearable Devices

Kyle Ross, Pritam Sarkar, Dirk Rodenburg, Aaron J. Ruberto, Paul Hungler, Adam Szulewski, Daniel Howes, Ali Etemad

TL;DR

This paper introduces a novel framework for dynamically adaptive simulations, specifically for trauma response training. By using wearable sensors to collect ECG and GSR data, they demonstrate that machine learning can accurately classify user expertise, allowing simulations to adjust their difficulty to optimize learning outcomes. This work paves the way for more personalized and effective training experiences.

What Problem Does This Paper Address?

Simulation-based training is a powerful tool, especially in high-stakes fields like emergency medicine. However, a significant challenge arises when the simulation's difficulty doesn't match the trainee's expertise level. If a simulation is too easy, it fails to challenge the learner and promote growth. Conversely, if it's too difficult, it can lead to frustration and negative learning outcomes. This mismatch, often referred to as the "expertise reversal effect," hinders effective learning. The authors aim to solve this by creating a system that can autonomously assess a user's expertise in real-time and dynamically adjust the simulation's complexity to ensure optimal engagement and learning.

Methods

The core of this research lies in using physiological signals from wearable devices to infer a user's expertise. Here's a breakdown of their approach:

  • Data Acquisition: They developed two realistic trauma simulation scenarios (penetrating and blunt force trauma) and collected data from both novice and expert trauma responders. During these simulations, participants wore Shimmer3 wearable sensors to record Electrocardiogram (ECG) and Galvanic Skin Response (GSR) signals.
  • Signal Pre-processing: Raw ECG signals were processed using the Pan-Tompkins algorithm to detect QRS complexes, with filters applied to reduce noise. GSR signals were also filtered.
  • Feature Extraction: A comprehensive set of features were extracted from both ECG (time and frequency domain) and GSR (time and frequency domain) signals. These features capture various aspects of physiological responses.
  • Feature Space Exploration: To visualize the separability of expert and novice data, they employed t-Distributed Stochastic Neighbor Embedding (t-SNE). This technique helps understand how well the extracted features can distinguish between the two groups.
  • Feature Selection: The Least Absolute Shrinkage and Selection Operator (LASSO) regression was used to identify the most discriminative features for expertise classification. LASSO penalizes less important features, effectively selecting the most relevant ones.
  • Classification: Several machine learning classifiers were trained and evaluated: Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and K-Nearest Neighbors (KNN). They tested these classifiers using unimodal (ECG only, GSR only) and multimodal (ECG + GSR) feature sets.
  • Evaluation: The performance of the classifiers was assessed using a Leave-One-Subject-Out (LOSO) cross-validation scheme, reporting accuracy and F1 scores.

Datasets

The paper does not introduce a new, publicly available benchmark dataset. Instead, they collected their own dataset during controlled trauma simulation scenarios. This dataset consists of:

  • ECG and GSR signals from 10 participants (5 experts, 5 novices) during two distinct trauma simulation scenarios.
  • Baseline physiological data collected from participants in a relaxed state before the simulation.

This custom dataset was specifically designed to capture physiological responses during a high-stakes, simulated medical emergency, allowing for the investigation of expertise-based differences.

Key Insights and Ideas

  • Physiological Signals as Expertise Indicators: The most significant insight is that physiological signals like ECG and GSR, captured by wearable devices, can indeed be used to reliably differentiate between novice and expert trauma responders. This moves beyond subjective assessments or performance metrics alone.
  • Baseline Correction is Crucial: The t-SNE visualizations clearly showed that baseline correction of the physiological signals significantly improved the separability of expert and novice feature spaces. This highlights the importance of accounting for individual resting physiological states.
  • Multimodal Fusion Outperforms Unimodal: Combining ECG and GSR features (multimodal approach) generally led to better classification performance compared to using either signal individually. This suggests that different physiological modalities capture complementary information about expertise.
  • ECG Features Show Strong Discriminative Power: While multimodal approaches were best overall, LASSO analysis indicated that ECG features, particularly those in the low-frequency bands, often had higher regression coefficients, suggesting they are highly important for distinguishing expertise.
  • Adaptive Simulations are Feasible: The successful classification of expertise using machine learning with wearable data demonstrates the practical feasibility of developing dynamically adaptive simulations that can tailor their difficulty to the user.

Key Results

  • Classification Accuracy: The multimodal feature set (ECG + GSR) achieved the highest classification performance. Using the KNN classifier with this feature set, they reported an accuracy of 0.8296 and an F1 score of 0.7996 using LOSO validation.
  • Feature Importance: LASSO analysis identified key features. For ECG, RRmean and ULF had high regression coefficients. For GSR, time-domain features like minimum HRT and standard deviation of RT were found to be important.
  • t-SNE Visualization: Baseline-corrected ECG, GSR, and multimodal features showed improved separability between expert and novice groups in the t-SNE plots, visually confirming the effectiveness of the pre-processing and feature extraction steps.

Limitations and Future Work

  • Binary Expertise Levels: The study only classified participants into two broad categories: expert and novice. The authors acknowledge that a more granular classification of expertise levels (e.g., intermediate levels) would provide a more accurate representation and allow for finer tuning of simulations.
  • External Review for Granularity: To achieve more granular expertise levels, they suggest using external reviewers to grade participant performance, providing a more objective measure.
  • Investigating Visual Complexity: Future work will explore how adding visual complexity (e.g., through AR elements) to simulations affects cognitive load and expertise classification.
  • Optimal Cognitive Load: Determining the optimal level of cognitive load for participants to achieve specific learning objectives is a key area for future research to maximize the effectiveness of adaptive simulations.
  • Generalizability: While promising, the study was conducted in a specific trauma simulation context. Further research would be needed to validate these findings across different simulation domains and tasks.

Summary updated

Abstract

Simulation-based training has been proven to be a highly effective pedagogical strategy. However, misalignment between the participant’s level of expertise and the difficulty of the simulation has been shown to have significant negative impact on learning outcomes. To ensure that learning outcomes are achieved, we propose a novel framework for adaptive simulation with the goal of identifying the level of expertise of the learner, and dynamically modulating the simulation complexity to match the learner’s capability. To facilitate the development of this framework, we investigate the classification of expertise using biological signals monitored through wearable sensors. Trauma simulations were developed in which electrocardiogram (ECG) and galvanic skin response (GSR) signals of both novice and expert trauma responders were collected. These signals were then utilized to classify the responders’ expertise, successive to feature extraction and selection, using a number of machine learning methods. The results show the feasibility of utilizing these bio-signals for multimodal expertise classification to be used in adaptive simulation applications.
BibTeX
@inproceedings{ross2019toward,
  author = {Kyle Ross and Pritam Sarkar and Dirk Rodenburg and Aaron J. Ruberto and Paul Hungler and Adam Szulewski and Daniel Howes and Ali Etemad},
  title = {{Toward Dynamically Adaptive Simulation: Multimodal Classification of User Expertise Using Wearable Devices}},
  booktitle = {Sensors},
  year = {2019},
  url = {https://openreview.net/forum?id=hACRf0ZGGzf},
}
Figure