Abstract
Machine learning plays an increasing role in intelligent tutoring systems as both the amount of data available and specialization among students grow. Nowadays, these systems are frequently deployed on mobile applications. Users on such mobile education platforms are dynamic, frequently being added, accessing the application with varying levels of focus, and changing while using the service. The education material itself, on the other hand, is often static and is an exhaustible resource whose use in tasks such as problem recommendation must be optimized. The ability to update user models with respect to educational material in real-time is thus essential; however, existing approaches require time-consuming re-training of user features whenever new data is added. In this paper, we introduce a neural pedagogical agent for real-time user modeling in the task of predicting user response correctness, a central task for mobile education applications. Our model, inspired by work in natural language processing on sequence modeling and machine translation, updates user features in real-time via bidirectional recurrent neural networks with an attention mechanism over embedded question-response pairs. We experiment on the mobile education application SantaTOEIC, which has 559k users, 66M response data points as well as a set of 10k study problems each expert-annotated with topic tags and gathered since 2016. Our model outperforms existing approaches over several metrics in predicting user response correctness, notably out-performing other methods on new users without large question-response histories. Additionally, our attention mechanism and annotated tag set allow us to create an interpretable education platform, with a smart review system that addresses the aforementioned issue of varied user attention and problem exhaustion.
Introduction
Mobile education platforms face unique challenges compared to traditional classroom settings. Users are highly dynamic, often engaging in short, distracted sessions, and the user base grows rapidly. Conversely, high-quality educational content is a finite resource. To be effective, intelligent tutoring systems must provide real-time updates to user profiles and handle the "cold-start" problem for new users. Traditional methods like Collaborative Filtering often struggle with these requirements due to the need for periodic retraining. This paper proposes a neural pedagogical agent designed to address these issues through real-time sequence modeling and interpretable attention mechanisms.
Methodology
The proposed architecture models the sequence of a user's question-response pairs to predict future performance.
Key components of the model include:
- Embedding Layer: Questions and responses are mapped into a shared latent space, allowing the model to learn semantic relationships between different problems.
- Bidirectional LSTM (Bi-LSTM): This captures temporal dependencies in the learning history. The backward LSTM is particularly effective at anchoring current performance against early foundational interactions.
- Additive Attention: Instead of a simple average of past performance, the model uses an attention mechanism to weight previous interactions based on their relevance to the current question being assessed.
- Real-Time Inference: By mapping users into a latent space through the Bi-LSTM, the system can update user features instantly as new data arrives without requiring a full system retrain.
Experimental Setup
The model was evaluated using the SantaTOEIC dataset, which represents a large-scale real-world application for English learning. The dataset includes 559,000 users and 66 million response data points. As shown in the figure below, the user base expanded significantly during the study period while the question pool remained relatively stable, emphasizing the need for efficient user modeling over static content.
Results and Discussion
The neural pedagogical agent demonstrated superior performance across multiple metrics compared to baselines like Matrix Factorization and Recurrent Recommender Networks (RRN). The model achieved an F1-score of 0.8114 and an AUC of 0.7661.
Crucially, the model excels in "cold-start" scenarios. It reaches an F1-score of 0.7 after only 44 questions, significantly faster than competing methods. This efficiency is vital for maintaining user engagement on mobile platforms where early drop-off rates can be high.
Beyond predictive accuracy, the attention mechanism provides valuable interpretability. The learned attention weights were found to align closely with expert-defined topic tags. This allows for a "Smart Review" system: by identifying past incorrect answers that the model is currently "attending" to, the system can recommend specific problems for review that are most likely to improve the student's mastery of the current topic.
Conclusion
By jointly learning to review past performance and assess future outcomes, the proposed neural pedagogical agent provides a scalable, real-time solution for mobile education. Future work will explore the integration of these predictive models directly into recommendation logic, moving from performance assessment to active learning path optimization using Transformer-based architectures.
Summary updated