Online Classroom Student Engagement Analysis using Enhanced YOLOv5

Shuai Wang, Abdul Samad Shibghatullah, Javid Iqbal, Kay Hooi KeoyProceedings of International Conference on Artificial Life and Robotics 2024 · DOI

TL;DR

Researchers have developed SFER-YOLOv5, an optimized deep learning model designed to monitor student engagement in virtual classrooms in real-time. By enhancing the popular YOLOv5 architecture with coordinate attention and improved loss functions, the system achieves higher accuracy in recognizing facial expressions, providing a proactive tool to detect student distress and potential cyberbullying.

What Problem Does This Paper Address?

The shift to online education following the COVID-19 pandemic created a "black box" for educators: it is significantly harder to gauge student understanding and emotional well-being through a computer screen than in a physical classroom. This lack of visibility contributes to two major issues:

  1. Disengagement: Teachers struggle to identify when students are confused or "checked out."
  2. Cyberbullying: Virtual classrooms are susceptible to online harassment, which can severely impact a student's psychological state and learning outcomes.

Current facial expression recognition (FER) tools often struggle with real-time performance or lose accuracy in "crowded" digital environments where students may be at varying distances from their cameras. The authors sought to bridge this gap by creating a fast, accurate detection system tailored for the classroom.

Methods

The authors built their solution on YOLOv5s, the "small" and fastest variant of the YOLOv5 object detection family. To make it fit for classroom emotional analysis, they introduced three "transformative optimizations":

  • Coordinate Attention (CA) Module: Standard attention mechanisms often ignore positional information. The CA module incorporates spatial coordinates into the channel attention, helping the model "focus" better on faces even when students are in the back rows or far from the camera.
  • Soft NMS (Non-Maximum Suppression): Traditional NMS aggressively deletes overlapping detection boxes. Soft NMS instead decays the confidence scores of overlapping boxes, which is more effective at detecting multiple students sitting close together.
  • Improved Loss Function (EIOU + Focal Loss): They replaced the standard box regression loss with Efficient-IOU (EIOU) to better handle width and height calculations. They also added Focal Loss to address "sample imbalance"—the fact that some emotions (like neutral) appear much more frequently in data than others (like sadness).

Datasets

The model was trained and validated using two primary sources:

  1. Fer2013: A well-known public dataset. The authors cleaned and filtered this down to 11,000 high-quality images to improve experimental accuracy.
  2. Self-Constructed Classroom Dataset: To ensure real-world applicability, the team collected 668 images from actual classroom video monitoring. These frames were manually annotated to capture the specific nuances of student expressions during a lecture.

Key Insights and Ideas

  • The Emotion-Mastery Link: The paper operates on the premise that student emotions are proxies for learning. Positive emotions (happiness/surprise) signal active engagement, while negative emotions (sadness/anger/disgust) often indicate a lack of concentration or a dislike of the material.
  • Spatial Awareness Matters: One of the "aha" moments in the paper is the realization that standard object detectors struggle with depth in a classroom. By adding the Coordinate Attention module, the network acquires more detailed information about students regardless of their seating position.
  • Proactive vs. Reactive: Instead of waiting for a student to report bullying or failure, this system aims for "early detection," allowing teachers to adjust their teaching mode or intervene the moment emotional irregularities appear.

Key Results

The SFER-YOLOv5 model outperformed the baseline YOLOv5 across the board:

  • Self-Constructed Dataset: The model saw a 3.9% increase in detection accuracy (P) and a 3.0% increase in mean Average Precision ([email protected]), reaching an overall accuracy of 87.3%.
  • Fer2013 Dataset: Accuracy improved from 73.1% to 75.6%.
  • Specific Emotions: The model was particularly effective at identifying "happy" and "sad" expressions, which are critical indicators of student well-being.
  • Efficiency: By using the YOLOv5s backbone, the system maintains the high inference speeds (up to 140 FPS) required for real-time video monitoring.

Limitations and Future Work

While the results are promising, the study has a few areas for growth:

  • Expression Variety: The primary comparative analysis focused on three main expressions: happiness, sadness, and neutrality. While these are key, they don't capture the full spectrum of human emotion.
  • Contextual Cues: Facial expressions alone don't tell the whole story. The authors acknowledge that future research should integrate poses and gestures (e.g., a student putting their head on the desk) to provide a more holistic analysis of classroom dynamics.
  • Privacy and Ethics: While not the focus of the technical paper, the deployment of such systems in real classrooms would require careful consideration of student privacy and data security.

Summary updated

Abstract

The surge in online education has underscored the pressing issue of cyberbullying in virtual classrooms.This paper introduces an inventive method for early cyberbullying detection by analyzing students' engagement and emotional responses in online classrooms.The proposed SFER-YOLOv5 model, integrating Student Facial Expression Recognition with an enhanced YOLOv5 object detection model, incorporates transformative optimizations.These include Soft NMS for Non-Maximum Suppression, the integration of a Channel Attention (CA) module within the YOLOv5 architecture, and the use of Enhanced Intersection over Union (EIOU) as the bounding box regression loss function.This approach identifies diminished engagement and emotional irregularities, offering a proactive framework for mitigating cyberbullying in online classrooms.
BibTeX
@inproceedings{wang2024online,
  author = {Shuai Wang and Abdul Samad Shibghatullah and Javid Iqbal and Kay Hooi Keoy},
  title = {{Online Classroom Student Engagement Analysis using Enhanced YOLOv5}},
  booktitle = {Proceedings of International Conference on Artificial Life and Robotics},
  year = {2024},
  doi = {10.5954/icarob.2024.gs3-1},
}
Figure