Abstract
In this paper, we take a preliminary step towards solving the problem of causal discovery in knowledge tracing, i.e., finding the underlying causal relationship among different skills from real-world student response data. This problem is important since it can potentially help us understand the causal relationship between different skills without extensive A/B testing, which can potentially help educators to design better curricula according to skill prerequisite information. Specifically, we propose a conceptual solution, a novel causal gated recurrent unit (GRU) module in a modified deep knowledge tracing model, which uses i) a learnable permutation matrix for causal ordering among skills and ii) an optionally learnable lower-triangular matrix for causal structure among skills. We also detail how to learn the model parameters in an end-to-end, differentiable way. Our solution placed among the top entries in Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. We detail preliminary experiments as evaluated on the challenge's public leaderboard since the ground truth causal structure has not been publicly released, making detailed local evaluation impossible.
Introduction
Knowledge Tracing (KT) is the task of modeling a student's mastery of skills over time based on their past performance. While deep learning models like Deep Knowledge Tracing (DKT) are excellent at predicting whether a student will get the next question right, they usually treat skills as a "flat" list. They fail to capture the underlying causal structure—for example, the fact that a student must master "Counting" before they can understand "Addition."
Traditionally, these prerequisite maps are built by human experts, which is time-consuming and hard to scale. Discovering these relationships automatically from observational data (student logs) is a "causal discovery" problem. The challenge is making this process end-to-end and differentiable so that the model can learn the causal graph and the student's knowledge state simultaneously.
Proposed Methodology
The authors propose a Causal Gated Recurrent Unit (GRU). The core innovation lies in how they mask the weight matrices of the GRU to enforce a causal structure.
The Causal Mask
The model defines a mask $M = PLP^T$ to enforce a Directed Acyclic Graph (DAG) structure among skills. This decomposition allows the model to independently learn the sequence of skills and the density of their connections.
- $P$ (Permutation Matrix): Learns the causal ordering (which skill comes first).
- $L$ (Lower-Triangular Matrix): Represents the causal dependency (which specific skills influence others).
Differentiable Discovery
Since permutation matrices are discrete and non-differentiable, the authors utilize the Sinkhorn operator. This relaxes the matrix into a "doubly stochastic" matrix, allowing the model to learn the ordering via standard backpropagation. They use an adaptive strategy for the Sinkhorn operator, starting with a "soft" relaxation and gradually making it "harder" (closer to a true permutation) over the course of training.
Experimental Setup
The model was evaluated using the dataset from Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. The data, sourced from the Eedi online education platform, includes 1,855 unique skills (constructs) and 6,468 students responding to multiple-choice diagnostic math questions.
Key Results
The model was ranked among the top entries on the NeurIPS 2022 Challenge leaderboard. An ablation study highlighted the importance of high-dimensional embeddings and the adaptive training strategy:
| Model Variant | Leaderboard $F_1$ Score |
|---|---|
| No Embeddings / No Adaptive Strategy | 0.11 |
| No Embeddings / With Adaptive Strategy | 0.17 |
| 300D Embeddings / Adaptive Strategy | 0.33 |
| 300D Embeddings / Adaptive / Learnable $L$ | 0.43 |
Qualitatively, the model successfully recovered logical prerequisite chains that align with standard educational curricula.
Conclusion and Future Work
This work demonstrates that it is possible to discover causal prerequisite relationships in an end-to-end, differentiable manner. A major finding was that 300-dimensional skill embeddings nearly tripled the $F_1$ score, suggesting that the "identity" and "context" of a skill are as vital as the causal link itself. Future work should explore applying this causal module to more modern, attention-based (Transformer) KT models and developing "human-in-the-loop" systems where data-driven discovery assists human experts in refining curricula.
Summary updated