Tracing Mathematical Proficiency Through Problem-Solving Processes

Jungyang Park, Suho Kang, Jaewoo Park, Jaehong Kim, Jaewoo Shin, Seonjoon Park, Youngjae YuarXiv 2025 · DOI · arXiv

Introduction: Beyond Binary Outcomes

Traditional Knowledge Tracing (KT) serves as the backbone of Intelligent Tutoring Systems, predicting student success on future tasks based on historical performance. However, these models typically rely on "outcome-centric" data—simple binary indicators of whether an answer was correct or incorrect. This approach creates a "black box" where the underlying reasons for a student's failure remain hidden, and models struggle with "cold-start" scenarios where little historical data exists.

Comparison between traditional outcome-centric Knowledge Tracing (a) and the proposed process-aware approach (b).

To address these limitations, researchers have introduced StatusKT, a framework that shifts the focus from final answers to the Problem-Solving Process (PSP)—the step-by-step logic and scratchwork students produce while solving a problem. By analyzing the "how" instead of just the "what," StatusKT provides a more granular view of student learning.

The StatusKT Framework

StatusKT utilizes a three-stage Large Language Model (LLM) pipeline (powered by GPT-5) to transform unstructured handwritten notes into structured "Mathematical Proficiency" (MP) signals:

  1. Indicator Extraction (Teacher LLM): The AI acts as a teacher to generate a problem-specific rubric across four dimensions: Conceptual Understanding (CU), Strategic Competence (SC), Procedural Fluency (PF), and Adaptive Reasoning (AR).
  2. Response Generation (Student LLM): The AI "pretends" to be the student, answering the teacher's rubric based solely on the OCR-transcribed handwritten notes.
  3. Proficiency Assessment (Teacher LLM): Finally, the AI evaluates those responses to calculate an MP Ratio—a score for each proficiency dimension.

These ratios are integrated into standard deep learning KT models (such as DKT or SAINT) as auxiliary signals. The system is trained using a composite loss function that forces the model to predict both the final answer and the student's underlying proficiency levels simultaneously.

The KT-PSP-25 Dataset

The study introduces KT-PSP-25, a first-of-its-kind dataset specifically designed for process-aware knowledge tracing. It contains 22,289 interactions from 1,343 students across 2,696 problems.

An example from the KT-PSP-25 dataset showing problem text, knowledge component routes, and the student's step-by-step solution process.

Unlike traditional datasets like ASSISTments or EdNet, which only track timestamps and correctness, KT-PSP-25 includes OCR-transcribed handwritten solution processes for every interaction. It also features rich metadata, including problem text, solution explanations, and 490 distinct Knowledge Components (KCs).

Key Insights and Results

The research highlights that raw text from student notes is often too noisy for direct model consumption due to OCR errors. However, distilling this text into structured MP ratios provides a "clean" intermediate signal that significantly enhances model performance.

  • Performance Gains: StatusKT consistently improved Area Under the Curve (AUC) and Accuracy (ACC) across ten different KT architectures. For example, on the DKT model, AUC rose from 0.6165 to 0.6197.
  • Cold-Start Robustness: The framework's most significant impact occurs when data is sparse. In scenarios with five or fewer interactions, process data reveals enough about a student's proficiency to make much better predictions about future performance than binary data ever could.
  • Interpretability: Beyond numerical improvements, StatusKT offers actionable insights for educators through proficiency "radar charts," identifying specific areas where a student may need targeted support.

Limitations and Future Directions

Despite its success, the framework faces several hurdles. OCR reliability remains an issue, with roughly 15% of transcriptions containing errors in complex mathematical notation. Furthermore, converting handwriting to text loses visual cues like diagrams and spatial arrows. Future work aims to expand this approach to other domains like physics and to improve the handling of multi-modal visual data to capture the full breadth of student reasoning.

Summary updated

Abstract

Knowledge Tracing (KT) aims to model student's knowledge state and predict future performance to enable personalized learning in Intelligent Tutoring Systems. However, traditional KT methods face fundamental limitations in explainability, as they rely solely on the response correctness, neglecting the rich information embedded in students' problem-solving processes. To address this gap, we propose Knowledge Tracing Leveraging Problem-Solving Process (KT-PSP), which incorporates students' problem-solving processes to capture the multidimensional aspects of mathematical proficiency. We also introduce KT-PSP-25, a new dataset specifically designed for KT-PSP. Building on this, we present StatusKT, a KT framework that employs a teacher-student-teacher three-stage LLM pipeline to extract students' Mathematical Proficiency (MP) as intermediate representation. In this pipeline, the teacher LLM first extracts problem-specific proficiency indicators, then a student LLM generates responses based on the student's solution process, and a teacher LLM evaluates these responses to determine mastery of each indicator. The experimental results on KT-PSP-25 demonstrate that StatusKT improves the prediction performance of existing KT methods. Moreover, StatusKT provides interpretable explanations for its predictions by explicitly modeling students' mathematical proficiency. Code is available \href{https://github.com/jungyangpark/KT-PSP-25}{here}.
BibTeX
@misc{park2025tracing,
  author = {Jungyang Park and Suho Kang and Jaewoo Park and Jaehong Kim and Jaewoo Shin and Seonjoon Park and Youngjae Yu},
  title = {{Tracing Mathematical Proficiency Through Problem-Solving Processes}},
  year = {2025},
  eprint = {2512.00311},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.00311},
}
Figure