Classifying Learner Engagement through Integration of Multiple Data Sources

Carole R. Beal, Lei Qu, Hyokyeong LeeAAAI 2006

Abstract

Traditional "click-stream" analysis often fails to capture the underlying psychological factors driving student interactions with educational software. This study integrates behavioral logs from an Intelligent Tutoring System (ITS) with self-reported motivation profiles and teacher evaluations to classify learner engagement. We find that a student's motivational state is a primary determinant of their learning strategy, often outweighing prior academic achievement.

Introduction

Intelligent Tutoring Systems (ITS) offer personalized learning but are susceptible to "gaming the system"—behaviors like help abuse or random guessing intended to bypass learning. While temporal data can identify when disengagement occurs, it rarely explains why. Understanding these psychological drivers is essential for developing software that provides appropriate pedagogical interventions. This research bridges the gap by combining behavioral data with psychological profiling.

Methodology

The study involved 83–91 high school students using "Wayang Outpost," a math ITS. Engagement was analyzed through four primary lenses:

  1. Behavioral Action Patterns: Finite-State Machines (FSMs) defined five patterns: Independent-a (fast/correct), Independent-b (correct after error), Guessing, Help Abuse, and Learning (processing hints).
  2. Motivation Profiles: Students completed surveys measuring math self-efficacy, interest, and perceived difficulty.
  3. Teacher Ratings: Teachers assessed student achievement and motivation levels.
  4. Cluster Analysis: Hierarchical clustering grouped students into three motivation profiles: Average (Group 1), Low (Group 2), and High (Group 3).
Table 1: Mean scores of psychological constructs defining the three student motivation clusters.

Results

The FSM approach successfully categorized over 90% of problem events. Analysis revealed that 53% of problems were solved independently, 22% involved genuine learning from hints, and 20% were "gamed."

Crucially, behavioral patterns varied significantly by motivation cluster ($p < .001$). As shown in Table 2, students in the Low Motivation group (Group 2) were the most likely to engage in "Learning" behaviors, while the Average Motivation group (Group 1) showed the highest tendency for "Guessing."

Table 2: Distribution of problem-solving action patterns across the three motivation groups.

Discussion

Motivation vs. Achievement

A student’s self-reported motivation proved to be a stronger predictor of ITS usage than prior math achievement. Even high-achieving students may resort to gaming behaviors if their motivation is low or if they possess a "fixed mindset" regarding their abilities.

The Software as a "Safe Space"

A surprising finding was that low-motivation, low-efficacy students (Group 2) utilized the "Learning" strategy (hint processing) more frequently than their peers. This suggests that the ITS provides a private, non-judgmental environment that encourages help-seeking in students who might otherwise feel anxious or discouraged in a traditional classroom setting.

Teacher Perception vs. Algorithmic Detection

While teachers were effective at identifying students who could solve problems independently, they frequently missed specific "gaming" behaviors like help abuse. This highlights the value of automated behavioral tracking in identifying subtle forms of disengagement.

Conclusion

Integrating psychological profiles with behavioral logs provides a more nuanced understanding of learner engagement. Future work will focus on using these profiles to adapt ITS feedback in real-time—for instance, by restricting hints for "gamers" or providing proactive encouragement for anxious learners.

Summary updated

BibTeX
@inproceedings{beal2006classifying,
  author = {Carole R. Beal and Lei Qu and Hyokyeong Lee},
  title = {{Classifying Learner Engagement through Integration of Multiple Data Sources}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure