Abstract
Traditional "click-stream" analysis often fails to capture the underlying psychological factors driving student interactions with educational software. This study integrates behavioral logs from an Intelligent Tutoring System (ITS) with self-reported motivation profiles and teacher evaluations to classify learner engagement. We find that a student's motivational state is a primary determinant of their learning strategy, often outweighing prior academic achievement.
Introduction
Intelligent Tutoring Systems (ITS) offer personalized learning but are susceptible to "gaming the system"—behaviors like help abuse or random guessing intended to bypass learning. While temporal data can identify when disengagement occurs, it rarely explains why. Understanding these psychological drivers is essential for developing software that provides appropriate pedagogical interventions. This research bridges the gap by combining behavioral data with psychological profiling.
Methodology
The study involved 83–91 high school students using "Wayang Outpost," a math ITS. Engagement was analyzed through four primary lenses:
- Behavioral Action Patterns: Finite-State Machines (FSMs) defined five patterns: Independent-a (fast/correct), Independent-b (correct after error), Guessing, Help Abuse, and Learning (processing hints).
- Motivation Profiles: Students completed surveys measuring math self-efficacy, interest, and perceived difficulty.
- Teacher Ratings: Teachers assessed student achievement and motivation levels.
- Cluster Analysis: Hierarchical clustering grouped students into three motivation profiles: Average (Group 1), Low (Group 2), and High (Group 3).
Results
The FSM approach successfully categorized over 90% of problem events. Analysis revealed that 53% of problems were solved independently, 22% involved genuine learning from hints, and 20% were "gamed."
Crucially, behavioral patterns varied significantly by motivation cluster ($p < .001$). As shown in Table 2, students in the Low Motivation group (Group 2) were the most likely to engage in "Learning" behaviors, while the Average Motivation group (Group 1) showed the highest tendency for "Guessing."
Discussion
Motivation vs. Achievement
A student’s self-reported motivation proved to be a stronger predictor of ITS usage than prior math achievement. Even high-achieving students may resort to gaming behaviors if their motivation is low or if they possess a "fixed mindset" regarding their abilities.
The Software as a "Safe Space"
A surprising finding was that low-motivation, low-efficacy students (Group 2) utilized the "Learning" strategy (hint processing) more frequently than their peers. This suggests that the ITS provides a private, non-judgmental environment that encourages help-seeking in students who might otherwise feel anxious or discouraged in a traditional classroom setting.
Teacher Perception vs. Algorithmic Detection
While teachers were effective at identifying students who could solve problems independently, they frequently missed specific "gaming" behaviors like help abuse. This highlights the value of automated behavioral tracking in identifying subtle forms of disengagement.
Conclusion
Integrating psychological profiles with behavioral logs provides a more nuanced understanding of learner engagement. Future work will focus on using these profiles to adapt ITS feedback in real-time—for instance, by restricting hints for "gamers" or providing proactive encouragement for anxious learners.
Summary updated