Revisiting Applicable and Comprehensive Knowledge Tracing in Large-Scale Data

Yiyun Zhou, Wenkang Han, Jingyuan Chen

Introduction

Knowledge Tracing (KT) serves as the foundational engine for Intelligent Tutoring Systems (ITS), enabling software to model a student's evolving mastery of various concepts. By analyzing historical performance, KT models predict how a student will fare on future tasks, allowing for personalized learning paths. While Deep Learning-based Knowledge Tracing (DLKT) has significantly improved predictive accuracy over traditional methods, recent research has identified critical gaps in how these models are designed and evaluated.

This paper introduces DKT2, a next-generation KT model that leverages the high-performance xLSTM architecture. By synthesizing modern deep learning with classical educational psychology—specifically Item Response Theory (IRT) and Rasch models—DKT2 achieves state-of-the-art accuracy while maintaining the practical requirements for real-world educational deployment.

The Three Cracks in Current KT Research

Despite the proliferation of DLKT models, the authors identify three major limitations that hinder their utility in actual classroom settings:

  1. The Applicability Gap: Many high-performing models (such as AKT) rely on "weakly applicable" settings. These models often require knowing the specific question a student will answer next to make a prediction, or they inadvertently use future information during training. This makes them difficult to implement in live, real-time software where the next question might be determined by the model's own output.
  2. The Comprehensiveness Gap: Most existing models focus on "one-step-ahead" prediction—predicting the score for a single specific question. They often fail to provide a "comprehensive" knowledge state, which is a holistic map of a student’s proficiency across all possible concepts at any given moment.
  3. The LSTM Bottleneck: Early DLKT models relied on standard LSTMs, which suffer from limited memory capacity and an inability to process data in parallel. This makes them slow and less effective when dealing with the massive, long-term datasets generated by modern educational platforms.

Methodology: The DKT2 Framework

DKT2 addresses these gaps by revamping the KT pipeline through three key technical pillars:

1. The xLSTM (Extended LSTM) Engine

DKT2 utilizes the recently developed xLSTM architecture to overcome the limitations of traditional recurrent networks. It specifically employs:

  • sLSTM: Utilizes exponential gating to better revise and update memory as new information arrives.
  • mLSTM: Replaces scalar memory with matrix memory, allowing the model to store more complex relationships and process data in parallel. This architecture enables DKT2 to handle significantly longer student histories and larger datasets than its predecessors.

2. Rasch-Model Embeddings

To improve question representation, DKT2 integrates the Rasch model from educational psychology. This allows the model to explicitly account for question difficulty and the degree to which a specific item deviates from the general concept it is intended to measure. This adds a layer of psychometric rigor often missing from pure "black box" deep learning models.

3. IRT-based Interpretability

DKT2 incorporates Item Response Theory (IRT) to decompose learned knowledge into "familiar" and "unfamiliar" categories. By distinguishing between concepts a student has mastered and those they are still being introduced to, the model can more accurately fuse this information to predict future performance.

Crucially, DKT2 is designed to be strongly applicable (relying only on past data) and comprehensive (outputting a proficiency vector for every concept in the system at every step).

Experimental Setup and Results

The model was evaluated on three large-scale benchmark datasets: Assist17 (math-focused), EdNet (a massive dataset from a South Korean AI tutor), and Comp (computational thinking and programming). DKT2 was tested against 18 baseline models, including popular architectures like AKT, simpleKT, and Mamba4KT.

Key Findings

  • Superior Accuracy: DKT2 consistently outperformed all baselines across all three datasets in both AUC (Area Under the Curve) and Accuracy for one-step predictions.
  • Multi-Step Stability: When predicting performance 5, 10, or 20 steps into the future, DKT2 demonstrated superior stability. While most models suffer from rapid accuracy decay due to error accumulation, the xLSTM architecture effectively mitigated this trend.
  • Scalability: The performance gap between DKT2 and other models widened as student history length increased, proving its ability to utilize long-term data that other models tend to "forget."
  • The "Future Information" Trap: The research highlights that DKT2 achieves top-tier results without relying on future question IDs, proving that "cheating" with future data is unnecessary for high performance.

Limitations and Future Directions

While DKT2 represents a significant step forward, the authors note several areas for further exploration. Current datasets often lack "ground truth" proficiency scores for all concepts simultaneously, requiring models to rely on the assumption that knowledge changes gradually. Furthermore, the authors aim to test DKT2 on "ultra-large-scale" data involving millions of students to see if the advantages of the xLSTM architecture scale linearly. Future work will focus on refining multi-concept analysis to better validate how accurately these models map the nuances of human learning and forgetting.

Summary updated

BibTeX
@inproceedings{zhou2025revisiting,
  author = {Yiyun Zhou and Wenkang Han and Jingyuan Chen},
  title = {{Revisiting Applicable and Comprehensive Knowledge Tracing in Large-Scale Data}},
  booktitle = {ECML/PKDD},
  year = {2025},
  url = {https://openreview.net/forum?id=Za3QPTkGKx},
}
Figure