Priority Guided Explanation for Knowledge Tracing with Dual Ranking and Similarity Consistency

Fan Li, Tiancheng Zhang, Yifang Yin, Minghe Yu, Mengxiang Wang, Ge YuIJCAI 2025 · DOI · OpenReview

Abstract

Knowledge Tracing (KT) models are essential for personalized education, yet their "black-box" nature often limits their practical utility. Existing explanation methods typically provide binary importance labels that lack the granularity needed for actionable learning plans. This paper introduces a model-agnostic explanation method that generates a ranked priority of past student interactions. By integrating a Directional Ranking Loss with Similarity-Aware Attribution Consistency, the approach identifies the most influential interactions while ensuring pedagogical consistency across similar concepts. Evaluations on benchmark datasets (ASSISTment 2009 and 2017) demonstrate that this method significantly outperforms state-of-the-art baselines in identifying key features across various KT architectures.

Introduction: Beyond Binary Importance

Knowledge Tracing (KT) leverages a student's history of interactions to predict future performance, enabling personalized learning paths. While deep learning models like DKT and AKT have achieved high accuracy, they lack interpretability, which is crucial for educators and students.

Current explanation methods for KT generally fall into two categories: intrinsic methods and feature attribution methods. Intrinsic methods build interpretability into the model itself, often at the cost of predictive power. Feature attribution methods, such as LIME or SHAP, attempt to explain existing models by identifying "important" interactions. However, these often provide only binary classifications (important vs. unimportant) and fail to rank interactions by priority. Furthermore, they frequently ignore the pedagogical reality that interactions involving similar knowledge concepts should have similar importance.

Comparison of existing explanation methods with the proposed Priority Guided Explanation.

Proposed Method: Priority Guided Explanation

The authors propose a mask optimization approach that learns a vector of importance scores for a student's history without altering the underlying KT model. This makes the method model-agnostic and applicable to any architecture, including RNNs and Transformers. Two core innovations drive this approach:

  1. Directional Ranking Loss (DRLPE): Standard explainers often measure any change in output when features are removed. DRLPE specifically focuses on the direction of change. It assumes a ranking consistency: removing a high-priority interaction should degrade model performance more significantly than removing a low-priority one. This allows the model to generate a fine-grained, ordered list of interactions for review.
  2. Similarity-Aware Attribution Consistency: In education, temporal recency is not the only factor in importance; topic similarity matters. This method introduces a loss term ensuring that interactions involving similar knowledge concepts receive similar importance scores, preventing erratic or counter-intuitive explanations.
Comparison of the proposed method with standard Deletion and Preservation games in terms of performance impact.

Experimental Evaluation

The researchers evaluated their method using the ASSISTment 2009 and ASSISTment 2017 public benchmarks. They tested the explainer against four diverse KT models: DKT (RNN-based), DKVMN (Memory-based), SAKT (Transformer-based), and AKT (Attention-based).

Key Results

Compared against seven baselines, including LIME, KernelSHAP, and DynaMask, the proposed method demonstrated superior performance:

  • Comprehensiveness & Sufficiency: The method achieved significantly higher comprehensiveness (successfully identifying influential features) and lower sufficiency (confirming that unimportant features truly had minimal impact).
  • Fine-Grained Accuracy: In "removal experiments," where features are removed sequentially based on their ranked importance, the performance of the KT models dropped much more sharply when using the proposed ranking compared to other methods, validating the accuracy of the priority list.

Conclusion and Future Work

This work represents a shift from technical transparency to practical utility in Explainable AI (XAI) for education. By providing ranked, consistent priorities, the method enables the creation of personalized, actionable learning plans—telling a student not just what to review, but in what order.

While the method is computationally more intensive than simple gradient-based approaches, its accuracy offers significant benefits. Future research will focus on human-in-the-loop studies to determine if these ranked explanations lead to better learning outcomes in real-world classroom settings.

Summary updated

Abstract

Knowledge tracing plays a pivotal role in enabling personalized learning on online platforms. While deep learning-based approaches have achieved impressive predictive performance, their limited interpretability poses a significant barrier to practical adoption. Existing explanation methods primarily focus on specific model architectures and fall short in 1) explicitly prioritizing critical interactions to generate fine-grained explanations, and 2) maintaining similarity consistency across interaction importance. These limitations hinder actionable insights for improving student outcomes. To bridge the gap, we propose a model-agnostic approach that provides enhanced explanations applicable to diverse knowledge tracing methods. Specifically, we propose a novel ranking loss designed to explicitly optimize the importance ranking of past interactions by comparing their corresponding perturbed outputs. Furthermore, we introduce a similarity loss to capture temporal dependencies, ensuring consistency in the assigned importance scores for conceptually similar interactions. Extensive experiments conducted on various knowledge tracing models and benchmark datasets demonstrate substantial enhancements in explanation quality.
BibTeX
@inproceedings{li2025priority,
  author = {Fan Li and Tiancheng Zhang and Yifang Yin and Minghe Yu and Mengxiang Wang and Ge Yu},
  title = {{Priority Guided Explanation for Knowledge Tracing with Dual Ranking and Similarity Consistency}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/49},
}
Figure