Abstract
Knowledge Tracing (KT) is a critical component of personalized education, yet its development is often hindered by data privacy regulations that prevent the pooling of student data into centralized repositories. This paper explores an incremental learning approach using the Self-Attentive Knowledge Tracing (SAKT) algorithm to train models across multiple school datasets sequentially. By treating each school as a separate task within a continual learning framework, the researchers demonstrate that sequential training can achieve performance comparable to joint training on pooled data. The study highlights that the sequence of training and the inherent similarity between school datasets are key factors in model generalization and the mitigation of catastrophic forgetting.
Introduction
Knowledge Tracing (KT) aims to predict student performance on future tasks based on their historical interactions, serving as a cornerstone for personalized learning platforms. However, robust KT models typically require massive datasets, which are often locked in "data silos" due to privacy regulations like the Personal Data Protection Act (PDPA). This research investigates whether a model can be trained sequentially across different schools—learning from one institution and then moving to the next without re-accessing previous data—while maintaining high predictive accuracy and robustness.
Methods
The researchers adopted a Continual Learning (CL) framework utilizing the Self-Attentive Knowledge Tracing (SAKT) architecture. Unlike traditional Recurrent Neural Network (RNN) models, SAKT leverages the Transformer's attention mechanism to better handle the sparsity common in educational data. The experimental design focused on two primary training sequences, or "Scenarios," to evaluate how the order of exposure to different school environments affects the model's final intelligence.
Datasets and Data Distribution
The study utilized the ASSISTment 2009 benchmark dataset, specifically isolating data from three distinct schools (IDs: 1998, 5117, and 5049). While the number of learners was consistent across the schools (92–95 students), the volume of responses varied significantly, ranging from approximately 5,600 to over 19,000 interactions. Dimensionality reduction via t-SNE revealed that schools 1998 and 5049 shared similar data distributions, while school 5117 was identified as an outlier.
Key Insights and Results
The experiments yielded several critical findings regarding incremental knowledge tracing:
- The "Warm-Up" Effect: Training order significantly impacts performance. Scenario 2 (5049 → 5117 → 1998) proved superior, suggesting that starting with a larger, more diverse dataset (School 5049) provides a more stable foundation for generalization.
- Similarity and Stability: Model performance remained more stable when transitioning between schools with similar problem distributions. Transitions involving the outlier school (5117) often resulted in performance dips as the model adapted to a new "language" of student behavior.
- Implicit Recovery: Interestingly, the model could "recover" performance on earlier tasks if a later task in the sequence was similar to the first, indicating that shared representations across schools can help mitigate the classic AI challenge of catastrophic forgetting.
Overall, the incremental approach achieved performance levels "considerably similar" to joint training. For example, the AUROC trends in Scenario 2 closely matched those of the joint set. High AUPRC values (often 60-70%) further indicated the model's sensitivity to the imbalance between correct and incorrect answers in the dataset.
Conclusion and Future Work
This research demonstrates that incremental learning is a viable path for building effective KT models across decentralized school data without compromising privacy. While catastrophic forgetting remains a risk, strategic sequence planning and leveraging data similarity can preserve model performance. Future research will focus on implementing explicit regularization techniques like Elastic Weight Consolidation (EWC), scaling the framework to hundreds of schools, and incorporating more complex metadata to further refine the models.
Summary updated