TL;DR
This paper investigates how uncontrolled variables like task duration and engagement influence the accuracy of machine learning models predicting user activities from physiological signals. They found that task duration significantly impacts model performance, suggesting it might reflect individual differences rather than the intended activity, highlighting the need for careful experimental design in physiological signal studies.
What Problem Does This Paper Address?
Understanding user behavior and engagement with information access systems is crucial. Physiological signals (like heart rate, skin conductance, and pupil dilation) offer objective measures for this. However, laboratory user studies using these signals are notoriously sensitive to various factors. These "uncontrolled" or "confounding" variables, such as task sequence, duration, or even a participant's engagement level, can significantly affect the collected physiological data and, consequently, the insights derived from it. This paper aims to quantify the impact of these common, often overlooked, variables on the effectiveness of machine learning models used to interpret physiological signals during information processing tasks. This is important because if these uncontrolled variables heavily influence the signals, our interpretations of user engagement or cognitive load might be inaccurate, leading to flawed conclusions about system design.
Methods
The researchers conducted a pilot user study with 7 participants performing four distinct information processing activities (READ, LISTEN, SPEAK, WRITE). During these activities, they collected physiological data using an Empatica E4 wristband (for Electrodermal Activity - EDA and Blood Volume Pulse - BVP) and a Tobii Fusion eye-tracker (for Pupil Diameter - PD).
To analyze the data, they employed a machine learning approach. The core idea was to train a multi-class classifier to predict which of the four activities a participant was performing based on their physiological signals. They then performed an "ablation study" by incrementally adding potential confounding variables as features to the machine learning model. By observing how the model's classification accuracy changed with the addition of each variable, they could infer the variable's influence. They used a leave-one-participant-out cross-validation strategy to ensure robust evaluation.
Datasets
The study used data collected from their own pilot user study (N=7). No pre-existing public datasets were used for training or evaluation. The study design involved participants completing four types of information-processing activities (READ, LISTEN, SPEAK, and WRITE) under varying complexity levels (low and high).
Key Insights and Ideas
- Task Duration is a Major Confounder: The most significant finding is that task duration had a substantial impact on the machine learning model's performance. This suggests that duration might be capturing individual differences in how participants approach tasks (e.g., speed of reading or speaking) rather than directly reflecting the specific information processing activity itself.
- Engagement and Cumulative Time Spent Also Matter: While not as impactful as duration, 'engagement' and 'cumulative_time_spent' also showed notable influence on model performance, indicating that a participant's subjective engagement and accumulated fatigue can alter physiological signals.
- Complexity's Impact is Nuanced: The 'complexity' of the task had a less pronounced effect, and its influence was primarily seen in differentiating between READ and LISTEN activities, but not SPEAK or WRITE. This might be due to how complexity was defined differently across activity types.
- Task Sequence's Impact is Unclear (in this pilot): The 'task_sequence' variable did not show a significant impact on model performance in this pilot study. However, the authors acknowledge that the limited number of participants and task permutations might mask a true effect, suggesting further investigation with larger datasets.
- Methodology for Evaluating Experimental Design: The paper proposes a valuable methodology: using changes in ML model performance as a proxy to scrutinize and refine experimental designs for user studies involving physiological signals.
Key Results
- Model Performance: The study compared several machine learning models, with Naïve Bayes achieving the highest mean accuracy, followed closely by SVM, RF, and KNN. SVM was chosen for further analysis due to its effectiveness and non-probabilistic nature.
- Variable Impact on Accuracy (SVM):
- Adding 'complexity' as a feature resulted in a slight accuracy increase.
- Adding 'task_sequence' did not change accuracy but expanded the confidence intervals.
- Adding 'duration' significantly increased accuracy by about 10%.
- Adding 'engagement' increased accuracy by about 9% with a larger CI.
- Adding 'cumulative_time_spent' caused a substantial accuracy drop of 22%.
- Confusion Matrices: Visualizations of confusion matrices showed how the addition of variables shifted predictions, for example, 'duration' improving READ predictions but making SPEAK and LISTEN harder to distinguish.
Limitations and Future Work
- Small Sample Size: The study acknowledges its limitation due to a small sample size (N=7). This restricts the generalizability of the findings and might have masked the impact of variables like 'task_sequence'.
- Pilot Study: This was a pilot study, and further research with larger participant groups is needed to confirm these findings.
- Defining Complexity: The authors note that complexity was instantiated differently for different IPAs, which might explain its varied impact. Future work could explore more consistent definitions.
- Further Validation: For 'cumulative_time_spent', they suggest using auxiliary signals like eye-blink frequency and involuntary hand gestures to better validate its role as a proxy for fatigue.
- Refining Experimental Design: The authors emphasize the need for careful design and gauging of variables like task duration in future studies, potentially by informing users about expected durations and using timers.
Summary updated