Self-supervised Deep Learning for Reading Activity Classification

Md. Rabiul Islam, Shuji Sakamoto, Yoshihiro Yamada, Andrew Vargo, Motoi Iwata, Masakazu Iwamura, Koichi KisearXiv 2020 · OpenReview

TL;DR

This paper introduces a novel self-supervised deep learning (DL) approach for analyzing reading activities, which significantly outperforms traditional methods and fully-supervised DL, especially when labeled data is scarce. By leveraging unlabeled data for pre-training, this method effectively addresses the common challenge of limited annotated datasets in reading analysis, paving the way for more robust and efficient reading behavior classification systems.

What Problem Does This Paper Address?

Accurately understanding and analyzing human reading behavior is crucial for improving learning strategies and providing personalized feedback. However, a major bottleneck in applying powerful deep learning (DL) models to this domain is the scarcity of labeled data. Manually annotating reading activities, such as identifying when someone is reading, what they are reading, or their confidence level, is time-consuming and expensive. This lack of labeled data hinders the development of effective DL-based reading analysis systems, especially for "in-the-wild" scenarios where data collection is less controlled and more prone to noise.

Methods

The core contribution of this paper is a self-supervised deep learning (DL) method for reading activity classification. The approach involves two main stages:

  1. Self-supervised Pre-training: This stage utilizes a large amount of unlabeled sensor data (from EOG glasses and eye-trackers). The model is trained on a "pretext task" where it learns to predict transformations applied to the input data. For example, it might be trained to identify if a signal was rotated, flipped, or scaled. This process allows the model to learn general representations of the sensor data without requiring human annotations.
  2. Target Task Training: After pre-training, the learned representations are fine-tuned on a smaller amount of labeled data for the specific reading activity classification task (e.g., reading detection or confidence estimation). This fine-tuning process adapts the general features learned during pre-training to the specific downstream task.

The authors compare their proposed self-supervised DL method against two baselines:

  • Fully-supervised DL: This involves training a DL model directly on labeled data without any pre-training.
  • Support Vector Machines (SVMs): A traditional machine learning approach that relies on hand-crafted features.

Datasets

The study utilizes two primary types of datasets for evaluation:

  • Reading Detection Datasets:
    • Labeled Dataset: The OPU_RD dataset, collected using JINS MEME EOG glasses, accelerometer, and gyroscope sensors. This dataset contains approximately 32,708 segments for "not reading" (NR), 5,340 for "reading English" (EN), 5,792 for "reading Japanese horizontal" (JH), and 5,798 for "reading Japanese vertical" (JV) after noise removal.
  • Unlabeled Dataset: Collected from 39 participants at a conference, totaling about 1,359 hours of EOG, ACC, and GYRO data. This dataset was segmented into 30-second intervals with 15-second overlap, resulting in 177,921 segments after noise removal.
  • Confidence Estimation Datasets:
    • Labeled Dataset: Collected from 20 Japanese university students answering multiple-choice questions (MCQs) using an eye-tracker. This dataset contains a summary of answered MCQs, with a significant skew towards "confident" answers.
  • Unlabeled Dataset: Collected from 80 Japanese high school students answering vocabulary MCQs, totaling 57,460 samples.

Key Insights and Ideas

  • Self-supervised learning is a powerful solution for data-scarce domains: The paper strongly demonstrates that self-supervised pre-training on unlabeled data can significantly boost performance, especially when labeled data is limited. This is a crucial finding for many real-world applications where obtaining large labeled datasets is impractical.
  • Generalizability across different sensors and tasks: The proposed method shows effectiveness on two distinct reading-related tasks (reading detection and confidence estimation) using different sensor modalities (EOG glasses and eye-trackers). This suggests the approach is generalizable and not tied to a specific sensor or task.
  • "In-the-wild" data is challenging but manageable: The study uses datasets collected in real-world settings without strict restrictions, highlighting the robustness of the proposed method in handling noisy and varied data.
  • Distinguishing subtle reading behaviors: For reading detection, the self-supervised DL method was particularly effective at differentiating between "reading English" and "reading horizontally written Japanese," which are often confused due to similar horizontal eye movements. This indicates the model learns finer-grained features.

Key Results

  • Reading Detection: The proposed self-supervised DL method consistently outperformed both fully-supervised DL and SVM across all tested numbers of labeled training samples. It was particularly superior when training data was scarce (e.g., 50 samples per class), achieving significantly higher accuracy. Even with a large number of samples (5,340), it performed comparably to or better than fully-supervised DL.
  • For example, with 50 training samples per class, the self-supervised DL achieved significantly higher accuracy than fully-supervised DL and SVM.
  • Statistical analysis (ANOVA and post-hoc t-tests) confirmed the significant superiority of the self-supervised DL method.
  • Confidence Estimation: Similar trends were observed. The self-supervised DL method generally performed best, especially when labeled data was limited. While SVM was not always the worst, its performance was capped even with more data. The fully-supervised DL performance dropped significantly with insufficient samples.
  • With 200 samples per class, the self-supervised DL achieved approximately 70% accuracy, while fully-supervised DL and SVM were around 50-60%.
  • With 5,382 samples per class, self-supervised DL reached over 70% accuracy, outperforming the other methods.

Limitations and Future Work

  • Limitations:
    • EOG Noise: The EOG data in the reading detection dataset contained noise, which slightly impacted the pre-training accuracy for EOG compared to ACC and GYRO.
  • Difficult Cases in Confidence Estimation: Some cases of confident and unconfident responses were difficult to distinguish even with the proposed method, suggesting the need for additional sensor modalities.
  • Skewed Data in Confidence Estimation: The labeled dataset for confidence estimation had a significant skew towards confident answers, requiring specific data preparation techniques (oversampling).
  • Future Work:
    • Further Accuracy Improvement: The authors suggest exploring ways to further enhance accuracy, potentially by incorporating other sensors.
  • Application to Other Tasks: Extending the application of the proposed self-supervised DL method to other reading activity classification tasks and using different sensor types.
  • Facial Expression Analysis: For confidence estimation, the paper suggests that integrating camera data to capture facial expressions could provide valuable features for improved classification.

Summary updated

Abstract

Reading analysis can give important information about a user's confidence and habits and can be used to construct feedback to improve a user's reading behavior. A lack of labeled data inhibits the effective application of fully-supervised Deep Learning (DL) for automatic reading analysis. In this paper, we propose a self-supervised DL method for reading analysis and evaluate it on two classification tasks. We first evaluate the proposed self-supervised DL method on a four-class classification task on reading detection using electrooculography (EOG) glasses datasets, followed by an evaluation of a two-class classification task of confidence estimation on answers of multiple-choice questions (MCQs) using eye-tracking datasets. Fully-supervised DL and support vector machines (SVMs) are used to compare the performance of the proposed self-supervised DL method. The results show that the proposed self-supervised DL method is superior to the fully-supervised DL and SVM for both tasks, especially when training data is scarce. This result indicates that the proposed self-supervised DL method is the superior choice for reading analysis tasks. The results of this study are important for informing the design and implementation of automatic reading analysis platforms.
BibTeX
@misc{islam2020self,
  author = {Md. Rabiul Islam and Shuji Sakamoto and Yoshihiro Yamada and Andrew Vargo and Motoi Iwata and Masakazu Iwamura and Koichi Kise},
  title = {{Self-supervised Deep Learning for Reading Activity Classification}},
  year = {2020},
  eprint = {2012.03598},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.03598},
}
Figure