Use of Computer Vision Analysis for Labeling Inattention Periods in Eeg Recordings With Visual Stimuli

Dmitry Isaev, Samantha Major, Kimberly L. H. Carpenter, Jordan Grapel, Zhuoqing Chang, J. Matías Di Martino, David Carlson, Géraldine Dawson, Guillermo Sapiro

Abstract

Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children, and training on additional 2560 labeled frames (equivalent to 85.3 seconds of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984–0.993) and the median inter-rater reliability (Cohen’s kappa) with a trained human annotator was 0.888. Agreement with consensus annotation on four participants labeled independently by two human annotators was in the 0.827–0.960 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community.

Introduction

In neurodevelopmental research, particularly studies involving children with autism spectrum disorder (ASD), collecting high-quality Electroencephalography (EEG) data is a significant challenge. To ensure that recorded brain activity accurately reflects responses to visual stimuli, researchers must precisely identify periods when the participant is attending to the screen. Traditionally, this "attention coding" is performed manually by research assistants who watch hours of video frame-by-frame. This process is labor-intensive, subjective, and expensive. While high-end eye-tracking hardware offers an alternative, it is often cost-prohibitive and difficult to calibrate for young children. This study presents a scalable, objective, and low-cost pipeline using standard webcams and supervised machine learning to automate the detection of inattention.

Methods

The proposed system utilizes a multi-stage pipeline for automated labeling. First, computer vision analysis (CVA) is used to extract features from standard video recordings, including head pose (pitch, yaw, roll), eye gaze coordinates, and nose landmarks. To capture the directional nature of inattention (e.g., looking left vs. right), raw coordinates are transformed into "positive" and "negative" versions relative to the participant's median pose.

Table 1: Computer vision features extracted for the machine learning model, including directional transformations for nose, gaze, and head pose.

These features are fed into a Multi-Layer Perceptron (MLP) neural network trained to classify frames as "attention" or "inattention." A critical component of the methodology is a "human-in-the-loop" approach: a pre-trained model is fine-tuned for each new participant using a small batch of manually labeled frames (approximately 85 seconds of video) from their specific session. The researchers found that randomly sampling these frames from across the recording was significantly more effective than sequential sampling from the beginning of the video.

Results

The model was evaluated using a dataset of 23 children with ASD, comprising over 566,000 video frames. Using Leave-One-Subject-Out (LOSO) cross-validation and fine-tuning with 2,560 randomly sampled frames, the system achieved a median area under the receiver operating characteristic curve (AUC) of 0.989.

Furthermore, the system demonstrated human-level reliability. The median Cohen’s kappa for agreement between the model and a human annotator was 0.888. In comparisons involving consensus annotations from two independent human experts, the model's agreement (0.827–0.960) often exceeded the level of agreement between the humans themselves. This suggests the model provides a highly consistent and objective standard for data curation.

Discussion

This automated approach significantly reduces the human labor required for EEG data curation. Labeling the frames necessary for model fine-tuning takes roughly 45 minutes, after which the remainder of a multi-hour dataset can be processed automatically. By utilizing off-the-shelf cameras and providing a public graphical user interface (GUI), this technology is accessible to labs with limited budgets.

Limitations remain, particularly regarding face detection failures. In approximately 5.38% of frames, extreme head angles or occlusions prevented feature extraction. Future work may involve using interpolation to estimate attention during these gaps. Despite these challenges, the system represents a major step toward objective, scalable attention monitoring in clinical and research settings.

Summary updated

Abstract

<title>Abstract</title> Electroencephalography (EEG) recordings with visual stimuli require detailed coding to determine the periods of participant’s attention. Here we propose to use a supervised machine learning model and off-the-shelf video cameras only. We extract computer vision-based features such as head pose, gaze, and face landmarks from the video of the participant, and train the machine learning model (multi-layer perceptron) on an initial dataset, then adapt it with a small subset of data from a new participant. Using a sample size of 23 autistic children, and training on additional 2560 labeled frames (equivalent to 85.3 seconds of the video) of a new participant, the median area under the receiver operating characteristic curve for inattention detection was 0.989 (IQR 0.984–0.993) and the median inter-rater reliability (Cohen’s kappa) with a trained human annotator was 0.888. Agreement with consensus annotation on four participants labeled independently by two human annotators was in the 0.827–0.960 range. Our results demonstrate the feasibility of automatic tools to detect inattention during EEG recordings, and its potential to reduce the subjectivity and time burden of human attention coding. The tool for model adaptation and visualization of the computer vision features is made publicly available to the research community.
BibTeX
@inproceedings{isaev2025use,
  author = {Dmitry Isaev and Samantha Major and Kimberly L. H. Carpenter and Jordan Grapel and Zhuoqing Chang and J. Matías Di Martino and David Carlson and Géraldine Dawson and Guillermo Sapiro},
  title = {{Use of Computer Vision Analysis for Labeling Inattention Periods in Eeg Recordings With Visual Stimuli}},
  booktitle = {Scientific},
  year = {2025},
  doi = {10.1038/s41598-025-10511-2},
}
Figure