A Conceptual Model for End-to-End Causal Discovery in Knowledge Tracing

Nischal Ashok Kumar, Wanyong Feng, Jaewook Lee, Hunter McNichols, Aritra Ghosh, Andrew S. Lan

Abstract

In this paper, we take a preliminary step towards solving the problem of causal discovery in knowledge tracing, i.e., finding the underlying causal relationship among different skills from real-world student response data. This problem is important since it can potentially help us understand the causal relationship between different skills without extensive A/B testing, which can potentially help educators to design better curricula according to skill prerequisite information. Specifically, we propose a conceptual solution, a novel causal gated recurrent unit (GRU) module in a modified deep knowledge tracing model, which uses i) a learnable permutation matrix for causal ordering among skills and ii) an optionally learnable lower-triangular matrix for causal structure among skills. We also detail how to learn the model parameters in an end-to-end, differentiable way. Our solution placed among the top entries in Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. We detail preliminary experiments as evaluated on the challenge's public leaderboard since the ground truth causal structure has not been publicly released, making detailed local evaluation impossible.

Introduction

Knowledge Tracing (KT) is the task of modeling a student's mastery of skills over time based on their past performance. While deep learning models like Deep Knowledge Tracing (DKT) are excellent at predicting whether a student will get the next question right, they usually treat skills as a "flat" list. They fail to capture the underlying causal structure—for example, the fact that a student must master "Counting" before they can understand "Addition."

Traditionally, these prerequisite maps are built by human experts, which is time-consuming and hard to scale. Discovering these relationships automatically from observational data (student logs) is a "causal discovery" problem. The challenge is making this process end-to-end and differentiable so that the model can learn the causal graph and the student's knowledge state simultaneously.

Proposed Methodology

The authors propose a Causal Gated Recurrent Unit (GRU). The core innovation lies in how they mask the weight matrices of the GRU to enforce a causal structure.

The Causal Mask

The model defines a mask $M = PLP^T$ to enforce a Directed Acyclic Graph (DAG) structure among skills. This decomposition allows the model to independently learn the sequence of skills and the density of their connections.

Decomposition of the causal mask into permutation and triangular matrices to derive skill ordering.
  1. $P$ (Permutation Matrix): Learns the causal ordering (which skill comes first).
  2. $L$ (Lower-Triangular Matrix): Represents the causal dependency (which specific skills influence others).

Differentiable Discovery

Since permutation matrices are discrete and non-differentiable, the authors utilize the Sinkhorn operator. This relaxes the matrix into a "doubly stochastic" matrix, allowing the model to learn the ordering via standard backpropagation. They use an adaptive strategy for the Sinkhorn operator, starting with a "soft" relaxation and gradually making it "harder" (closer to a true permutation) over the course of training.

Experimental Setup

The model was evaluated using the dataset from Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. The data, sourced from the Eedi online education platform, includes 1,855 unique skills (constructs) and 6,468 students responding to multiple-choice diagnostic math questions.

Key Results

The model was ranked among the top entries on the NeurIPS 2022 Challenge leaderboard. An ablation study highlighted the importance of high-dimensional embeddings and the adaptive training strategy:

Model Variant Leaderboard $F_1$ Score
No Embeddings / No Adaptive Strategy 0.11
No Embeddings / With Adaptive Strategy 0.17
300D Embeddings / Adaptive Strategy 0.33
300D Embeddings / Adaptive / Learnable $L$ 0.43

Qualitatively, the model successfully recovered logical prerequisite chains that align with standard educational curricula.

Example of a recovered causal chain among educational constructs, moving from basic counting to volume units.

Conclusion and Future Work

This work demonstrates that it is possible to discover causal prerequisite relationships in an end-to-end, differentiable manner. A major finding was that 300-dimensional skill embeddings nearly tripled the $F_1$ score, suggesting that the "identity" and "context" of a skill are as vital as the causal link itself. Future work should explore applying this causal module to more modern, attention-based (Transformer) KT models and developing "human-in-the-loop" systems where data-driven discovery assists human experts in refining curricula.

Summary updated

Abstract

In this paper, we take a preliminary step towards solving the problem of causal discovery in knowledge tracing, i.e., finding the underlying causal relationship among different skills from real-world student response data. This problem is important since it can potentially help us understand the causal relationship between different skills without extensive A/B testing, which can potentially help educators to design better curricula according to skill prerequisite information. Specifically, we propose a conceptual solution, a novel causal gated recurrent unit (GRU) module in a modified deep knowledge tracing model, which uses i) a learnable permutation matrix for causal ordering among skills and ii) an optionally learnable lower-triangular matrix for causal structure among skills. We also detail how to learn the model parameters in an end-to-end, differentiable way. Our solution placed among the top entries in Task 3 of the NeurIPS 2022 Challenge on Causal Insights for Learning Paths in Education. We detail preliminary experiments as evaluated on the challenge's public leaderboard since the ground truth causal structure has not been publicly released, making detailed local evaluation impossible.
BibTeX
@inproceedings{kumar2023conceptual,
  author = {Nischal Ashok Kumar and Wanyong Feng and Jaewook Lee and Hunter McNichols and Aritra Ghosh and Andrew S. Lan},
  title = {{A Conceptual Model for End-to-End Causal Discovery in Knowledge Tracing}},
  booktitle = {EDM},
  year = {2023},
  url = {https://openreview.net/forum?id=58rxsu0Xi0},
}
Figure