A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics

Sharath Koorathota, Kaveri A. Thakoor, Linbi Hong, Yaoli Mao, Patrick Adelman, Paul SajdaFrontiers in Psychology 2021 · DOI

Abstract

There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal.

Introduction

For decades, pupillometry has served as a window into cognitive states such as mental effort, learning, and arousal. However, the "Pupillary Light Reflex" and other physical factors—including accommodation (focusing) and eye movements like saccades and blinks—introduce significant noise into the data. In real-world environments, such as a student watching an educational video, these non-cognitive factors can easily mask the subtle dilations associated with cognitive engagement. This research proposes a deep learning "filter" to isolate the cognitive signal by modeling and subsequently subtracting these physical components from the raw pupil data.

Methods

The researchers employed a Bidirectional Long Short-Term Memory (LSTM) network, a type of Recurrent Neural Network (RNN) designed to process sequence data over varying timescales. The model was trained to predict pupil diameter based on "physical" eye-tracking features that are typically associated with non-cognitive responses.

  • Input Features: The model utilized fixation positions (X, Y coordinates), fixation duration, saccades, and blink-related information.
  • AOI Embeddings: To provide environmental context, fixations were mapped to specific "Areas of Interest" (AOIs)—such as the instructor, text slides, or images—and converted into 50-dimensional embeddings. This allowed the model to account for luminance changes associated with different visual stimuli.
  • Experimental Design: 61 healthy subjects (ages 18–35) watched three 5-minute lecture videos. To ensure robustness, videos were presented in three modalities: Full (professor + slides), Dual (slides + audio), and Single (slides only).
Experimental setup including video selection, areas of interest (AOI), and data splitting strategy.

The data was processed using a sliding window approach. As illustrated below, the model used 10 seconds of input data to predict the pupil dynamics in the subsequent 5 to 10 seconds. The core hypothesis was that the residual (the difference between the actual pupil size and the model’s physical prediction) would represent the "unpredictable" cognitive effort.

Temporal windowing for the video data stream, showing the 10-second input and subsequent output windows.

Key Results

The LSTM model significantly outperformed standard linear regression and Random Forest models, achieving a mean Root-Mean-Square Error (RMSE) as low as 249.87 µm. Key findings include:

  • Predictability vs. Performance: The model exhibited lower error rates for lower-performing subjects. This suggests that the pupil dynamics of less-engaged students are driven primarily by physical factors (luminance/movement), whereas high-performers exhibit more "unpredictable" cognitive fluctuations.
  • Enhanced Correlation: The residuals of the LSTM model showed a higher correlation with post-viewing test scores ($r = 0.33$) than the raw, true pupil size ($r = 0.24$).
  • Classification Power: Using the residuals to classify subjects as "high" or "low" performers yielded an AUC of 0.74, which was notably higher than using raw pupil size (AUC = 0.65) or the model’s raw predictions (AUC = 0.63).

Discussion and Conclusion

This study demonstrates that deep recurrent models can effectively separate luminance- and movement-driven pupil responses from those linked to cognition and arousal. By treating the model's prediction as a baseline for physical noise, the resulting residual signal becomes a much more accurate index of learning performance.

While the LSTM remains a "black box" in terms of feature prioritization, its ability to enhance the signal-to-noise ratio in pupillometry opens doors for more accurate cognitive monitoring in uncontrolled, real-world settings. Future research should investigate whether this filtering approach remains effective with lower-quality data from standard webcams, which would facilitate broader applications in remote education and digital health.

Summary updated

Abstract

There is increasing interest in how the pupil dynamics of the eye reflect underlying cognitive processes and brain states. Problematic, however, is that pupil changes can be due to non-cognitive factors, for example luminance changes in the environment, accommodation and movement. In this paper we consider how by modeling the response of the pupil in real-world environments we can capture the non-cognitive related changes and remove these to extract a residual signal which is a better index of cognition and performance. Specifically, we utilize sequence measures such as fixation position, duration, saccades, and blink-related information as inputs to a deep recurrent neural network (RNN) model for predicting subsequent pupil diameter. We build and evaluate the model for a task where subjects are watching educational videos and subsequently asked questions based on the content. Compared to commonly-used models for this task, the RNN had the lowest errors rates in predicting subsequent pupil dilation given sequence data. Most importantly was how the model output related to subjects' cognitive performance as assessed by a post-viewing test. Consistent with our hypothesis that the model captures non-cognitive pupil dynamics, we found (1) the model's root-mean square error was less for lower performing subjects than for those having better performance on the post-viewing test, (2) the residuals of the RNN (LSTM) model had the highest correlation with subject post-viewing test scores and (3) the residuals had the highest discriminability (assessed via area under the ROC curve, AUC) for classifying high and low test performers, compared to the true pupil size or the RNN model predictions. This suggests that deep learning sequence models may be good for separating components of pupil responses that are linked to luminance and accommodation from those that are linked to cognition and arousal.
BibTeX
@inproceedings{koorathota2021recurrent,
  author = {Sharath Koorathota and Kaveri A. Thakoor and Linbi Hong and Yaoli Mao and Patrick Adelman and Paul Sajda},
  title = {{A Recurrent Neural Network for Attenuating Non-cognitive Components of Pupil Dynamics}},
  booktitle = {Frontiers in Psychology},
  year = {2021},
  doi = {10.3389/fpsyg.2021.604522},
}
Figure