Introduction: Beyond Binary Outcomes
Traditional Knowledge Tracing (KT) serves as the backbone of Intelligent Tutoring Systems, predicting student success on future tasks based on historical performance. However, these models typically rely on "outcome-centric" data—simple binary indicators of whether an answer was correct or incorrect. This approach creates a "black box" where the underlying reasons for a student's failure remain hidden, and models struggle with "cold-start" scenarios where little historical data exists.
To address these limitations, researchers have introduced StatusKT, a framework that shifts the focus from final answers to the Problem-Solving Process (PSP)—the step-by-step logic and scratchwork students produce while solving a problem. By analyzing the "how" instead of just the "what," StatusKT provides a more granular view of student learning.
The StatusKT Framework
StatusKT utilizes a three-stage Large Language Model (LLM) pipeline (powered by GPT-5) to transform unstructured handwritten notes into structured "Mathematical Proficiency" (MP) signals:
- Indicator Extraction (Teacher LLM): The AI acts as a teacher to generate a problem-specific rubric across four dimensions: Conceptual Understanding (CU), Strategic Competence (SC), Procedural Fluency (PF), and Adaptive Reasoning (AR).
- Response Generation (Student LLM): The AI "pretends" to be the student, answering the teacher's rubric based solely on the OCR-transcribed handwritten notes.
- Proficiency Assessment (Teacher LLM): Finally, the AI evaluates those responses to calculate an MP Ratio—a score for each proficiency dimension.
These ratios are integrated into standard deep learning KT models (such as DKT or SAINT) as auxiliary signals. The system is trained using a composite loss function that forces the model to predict both the final answer and the student's underlying proficiency levels simultaneously.
The KT-PSP-25 Dataset
The study introduces KT-PSP-25, a first-of-its-kind dataset specifically designed for process-aware knowledge tracing. It contains 22,289 interactions from 1,343 students across 2,696 problems.
Unlike traditional datasets like ASSISTments or EdNet, which only track timestamps and correctness, KT-PSP-25 includes OCR-transcribed handwritten solution processes for every interaction. It also features rich metadata, including problem text, solution explanations, and 490 distinct Knowledge Components (KCs).
Key Insights and Results
The research highlights that raw text from student notes is often too noisy for direct model consumption due to OCR errors. However, distilling this text into structured MP ratios provides a "clean" intermediate signal that significantly enhances model performance.
- Performance Gains: StatusKT consistently improved Area Under the Curve (AUC) and Accuracy (ACC) across ten different KT architectures. For example, on the DKT model, AUC rose from 0.6165 to 0.6197.
- Cold-Start Robustness: The framework's most significant impact occurs when data is sparse. In scenarios with five or fewer interactions, process data reveals enough about a student's proficiency to make much better predictions about future performance than binary data ever could.
- Interpretability: Beyond numerical improvements, StatusKT offers actionable insights for educators through proficiency "radar charts," identifying specific areas where a student may need targeted support.
Limitations and Future Directions
Despite its success, the framework faces several hurdles. OCR reliability remains an issue, with roughly 15% of transcriptions containing errors in complex mathematical notation. Furthermore, converting handwriting to text loses visual cues like diagrams and spatial arrows. Future work aims to expand this approach to other domains like physics and to improve the handling of multi-modal visual data to capture the full breadth of student reasoning.
Summary updated