Is Internal State Feedback in an E-Learning Environment Acceptable to People?

Atsushi Ashida, Ryosuke Kawamura, Shizuka Shirai, Noriko Takemura, Mehrasa Alizadeh, Hideaki Hayashi, Hajime NagaharaICCE 2024 · DOI

Introduction

On-demand e-learning has become a cornerstone of modern education, yet it faces a persistent challenge: high dropout rates driven by a lack of instructor-student interaction. Unlike traditional classrooms, instructors cannot observe student engagement or intervene when attention wavers. While computer vision can now estimate "internal states"—such as alertness or boredom—using standard webcams, the social acceptability of this technology remains under-explored. This research investigates the tension between the technical utility of automated engagement monitoring and the potential for it to be perceived as intrusive surveillance.

System Architecture and Methodology

The researchers developed a full-stack "Alertness Estimation System" designed to provide real-time feedback to learners. The system is divided into three primary components:

  1. The Client: A web application that captures facial video, displays learning content, and renders a real-time graph of the user's estimated alertness.
  2. The Model: A deep learning pipeline utilizing EfficientNet-B3 for spatial feature extraction and a Bidirectional Gated Recurrent Unit (BiGRU) to capture temporal dependencies over a 1-second window. The model focuses on the eye region to determine alertness levels.
  3. The Feedback Loop: The system attempts an estimation every second, updating a visualization graph that allows learners to objectively observe their own learning attitude.
Figure 1: The overview of the alertness estimation system and the CNN-BiGRU model architecture.

The model was trained on a dataset of 53 undergraduate students, specifically focusing on those who transitioned from alert to drowsy states while watching video lectures. In technical evaluations, the model achieved a robust average F1 macro score of 0.89.

Results and Sentiment Analysis

To evaluate social acceptance, the system was deployed at "The Lab" exhibition facility in Osaka, Japan, where 131 participants across various age groups provided qualitative feedback. The researchers performed a co-occurrence network analysis on free-text survey responses to identify key sentiments.

Figure 2: Co-occurrence network analysis of user feedback, highlighting the divide between user negativity and supervisor positivity.

The analysis revealed a significant "perspective divide":

  • User-Negative Sentiment: 35 responses were coded as negative. Learners frequently expressed feelings of being "scared," "judged," or "monitored." Many feared that such data could be weaponized, such as leading to pay cuts in a corporate setting.
  • Supervisor-Positive Sentiment: 22 responses were coded as positive, primarily from educators and parents. Instructors viewed the data as a valuable tool for identifying boring lecture segments and improving instructional design.

Discussion and Future Work

The study highlights that technical feasibility does not equate to social acceptability. While the AI is highly accurate, the "feeling of being monitored all the time" created a sense of aversion that often outweighed the perceived benefits. Furthermore, the high frequency of feedback (every second) was noted as potentially distracting, reducing the learner's sense of agency.

Future work will focus on optimizing feedback frequency and exploring "stealthier" delivery methods. The researchers also suggest shifting the focus toward a teacher dashboard, where the data serves as a diagnostic tool for instructors to improve content rather than a direct surveillance mirror for the student.

Summary updated

BibTeX
@inproceedings{ashida2024is,
  author = {Atsushi Ashida and Ryosuke Kawamura and Shizuka Shirai and Noriko Takemura and Mehrasa Alizadeh and Hideaki Hayashi and Hajime Nagahara},
  title = {{Is Internal State Feedback in an E-Learning Environment Acceptable to People?}},
  booktitle = {ICCE},
  year = {2024},
  doi = {10.58459/icce.2024.4817},
}
Figure