Introduction
Emotion recognition, particularly the estimation of user engagement in e-learning environments, remains a significant challenge for artificial intelligence. Human expressions are nuanced, culturally specific, and deeply contextual, often leading to performance plateaus for fully automated systems. Human-in-the-Loop (HITL) workflows address this by integrating human annotators to review and refine AI predictions. However, the psychological dynamics between the human and the machine in these systems are poorly understood. This paper investigates how actual model reliability and psychological framing—the perceived reliability of the AI—influence human trust, cognitive load, and correction behavior.
Methodology
Model Architecture
The researchers developed a baseline emotion recognition model designed to process video data. The architecture utilizes a hybrid approach: MobileNetV2 is employed to extract spatial features from individual video frames, while a Long Short-Term Memory (LSTM) network captures the temporal dynamics of emotional changes over time. This combination allows the system to understand both the static expression and the progression of the affective state.
Experimental Design
The study involved a behavioral experiment with approximately 30 participants across three distinct scenarios to test the limits of human-AI collaboration:
- S1 (Baseline): Participants interacted with the model's standard, accurate predictions.
- S2 (Fabricated Errors): The model's predictions were mathematically inverted (e.g., high engagement scores became low) to force human intervention.
- S3 (Negative Framing): The model provided accurate predictions (identical to S1), but participants were explicitly warned that the model was historically unreliable.
Dataset
The study utilized the DAiSEE dataset, a benchmark for emotion recognition in e-learning. It contains 9,068 ten-second webcam video snippets of students, annotated for four affective states: engagement, boredom, confusion, and frustration.
Key Results
Model Performance
The baseline MobileNetV2 + LSTM model demonstrated robust performance across the DAiSEE affective states. Specifically, for engagement detection, the model achieved an 86% F1-score and 80% accuracy, providing a strong foundation for the HITL experiments.
Human Annotation Behavior
- Trust and Agreement (S1): In the baseline scenario, human annotators showed high trust in the AI. Adjustments were minor and tightly clustered around the AI's original predictions.
- Error Correction (S2): When the AI was intentionally flawed, humans successfully identified and corrected the errors. However, this came at a high cost of cognitive strain and participant frustration, suggesting that humans cannot be used as a permanent safety net for low-quality AI without significant burnout.
- The Impact of Framing (S3): Negative framing significantly altered behavior. Despite the AI being accurate, participants became skeptical and made frequent, unnecessary corrections. Interestingly, 92% of those who heavily relabeled data in this scenario self-identified as having high critical thinking abilities.
Discussion and Insights
The findings highlight the "Goldilocks" zone of trust: reliable AI fosters high agreement and allows humans to focus on subtle refinements. Conversely, the "Aha!" moment of the study was the power of psychological framing. UI copy and onboarding instructions can be just as influential as the actual model accuracy in determining the quality of HITL data. Furthermore, demographic factors such as age and personality (e.g., critical thinking skills) play a significant role in how likely an annotator is to override machine suggestions.
Conclusion and Future Work
This research demonstrates that designing effective HITL systems requires balancing technical model reliability with human psychological factors. Future work should explore larger participant pools and multimodal models incorporating audio and physiological signals. Additionally, longitudinal studies are needed to understand how trust evolves over extended periods of human-AI collaboration.
Summary updated