Knowledge tracing is the task of predicting whether a student will answer the next question correctly from their past answers. Its deep versions are accurate but routinely break a basic fact of learning. On multi-skill math items, success behaves like a logical AND over the required skills: a student who lacks one of two needed skills does almost as badly as one who lacks both. Neural tracers have no built-in notion of this. They can predict that a student will solve "multi-step equations with fractions" while failing "fractions", and they fail most often in exactly the setting teachers care about, a single classroom with few students and few answers each. We propose a training loss that tells a knowledge tracer, for every pair of questions where one needs a strict superset of the other's skills, how much less likely the harder question should be for this particular student. The loss is a hinge with a margin. The margin is set from the model's own estimate of whether the student has the extra skills and from how much evidence backs that estimate.
The two-sentence version is this. A question that needs everything an easier question needs, plus more, should never be predicted easier. The gap should be large only when the student probably lacks the extra skills. Because the constraint can be applied to every question pair at every time step, including questions the student never answered, it supplies supervision without new labels. We expect it to help most when real labels are scarce. It also makes the model's predictions consistent with the AND-gate structure, so they can be read as prerequisite diagnoses.
Core Claim
The claim has three parts. The first is that a student-conditioned, evidence-scaled consistency margin over nested skill sets reduces how often predictions violate conjunctive structure, with no loss in predictive accuracy. The second is that its benefit over an unregularized model grows as the number of training students shrinks, because the constraint acts on unanswered item pairs. The third is that the margin must be dynamic. A fixed margin either over-penalizes students who have already mastered the extra skills, since for them the harder item is nearly as easy, or under-penalizes students who clearly lack them.
The empirical anchor is Pardos et al.. On ASSISTments 2005–06 multi-skill items, a Bayesian network allowed to learn any composition rule converged to a conjunctive one. With noise factored out, effective knowledge was about 0.05 with one of two skills mastered and about 1.0 with both. That paper used a static network and noted that the composition is underdetermined without longitudinal modeling. We move the conjunctive prior into a sequential neural model as a soft constraint rather than a fixed architecture.
Relation to Prior Work
The closest work is Question Difficulty Consistent Knowledge Tracing (QDCKT). QDCKT adds a pairwise penalty so that, within one skill, harder questions get lower predicted success. Our proposal differs from it in three ways. The ordering comes from skill-set containment across skills rather than from difficulty within one skill. The penalty has a margin that depends on the student and their current state rather than a constant. And the pairs are built across multi-parent structure, so they are not restricted to one skill tag.
Hierarchical Triplet Loss, from Deep Metric Learning with Hierarchical Triplet Loss, supplies the method idea. HTL sets an embedding-space margin by the level at which two image classes merge in a learned class tree. We borrow the principle that the margin should reflect structural distance. We apply it to predicted success probabilities rather than embeddings, and the structure is a known skill-containment order rather than a clustering that has to be recomputed. That removes the costly re-clustering step HTL requires.
The warm-up schedule we use is motivated by Bailer et al.. They observe that fixed margins over-constrain early training and stop giving signal late in training.
Three contrastive knowledge-tracing methods motivate the proposal by showing where current methods stop:
- Q-MCKT uses a static contrastive loss for rarely answered questions and averages concept states through mixture-of-experts gating. That averaging is a compensatory assumption, the opposite of the AND gate Pardos found.
- DCL4KT+LLM builds hard negatives by inverting predicted difficulty, but enforces no ordering constraint.
- L-HAKT aligns simulated and real student traces in hyperbolic space with a fixed contrastive objective. It also depends on an LLM to produce the hierarchy.
None of the three constrains predictions to respect skill containment.
For uncertainty, ReliCD shows that unconstrained cognitive diagnosis is badly overconfident, and we reuse its idea of confidence-aware mastery. For the low-resource regime, KT² shows that a hidden Markov tree beats deep tracers and LLMs when there are 50 students with 5 exercises each. That finding constrains us: a neural method must be shown to close this gap, not assumed to.
Research Program
The method has four components.
Where the pairs come from. Let $K_i$ be the set of skills tagged on item $i$. For each student $s$ at step $t$, we form ordered pairs $(i, j)$ with $K_i \subsetneq K_j$ and write $M = K_j \setminus K_i$ for the extra skills. These come directly from the item-to-skill tag matrix. Containment is well defined when a skill has several parents or an item has several skills, which avoids the single-parent-tree limitation noted for KT². As a secondary source, a prerequisite edge $u \to v$, whether expert-provided or taken from exercise-influence estimates as in Deep Knowledge Tracing, adds $u$ to the effective skill set of items tagged $v$.
The loss. Let $p_{s,t}(i)$ be the model's predicted probability that student $s$ answers item $i$ correctly at step $t$. The consistency term is
$$ \mathcal{L}_{\text{cc}} = \sum_{s,t}\sum_{(i,j)} \max\big(0,\; p_{s,t}(j) - p_{s,t}(i) + m_{s,t}(i,j)\big). $$
The margin is
$$ m_{s,t}(i,j) = \gamma\, w_{s,t}(M)\,\Big(1 - \prod_{k \in M} \hat q_{s,t}(k)\Big). $$
In the margin, $\hat q_{s,t}(k)$ is the model's current mastery estimate for skill $k$. It is read off as the predicted success on a representative single-skill item and passed through a stop-gradient, so the model cannot shrink its own margin by lowering these estimates. The weight $w_{s,t}(M) \in [0,1]$ grows with the number of observed responses on the skills in $M$, and it is multiplied by the confidence of $\hat q$ as estimated by Monte Carlo dropout. The constant $\gamma$ caps the largest gap.
We tie $\gamma$ to the data rather than tuning it freely. It is set no larger than one minus the guess rate minus the slip rate, both estimated on the training split. Pardos et al. report that guess and slip rates fall as skill count rises, so the cap loosens for items with more skills.
The product term is the AND gate. The weight $w$ addresses the feedback-loop risk flagged for difficulty-driven margins: when evidence is thin, the margin relaxes toward zero instead of confidently enforcing a wrong gap. Before $\gamma$ ramps up, the model trains only on the prediction loss for a warm-up period.
Label efficiency. Pairs are sampled over the whole item bank at every step, not only over answered items. The loss therefore supplies gradient on questions the student never saw. This plays the role that pretext tasks play in self-supervised learning, and we test it by varying how many students have labels.
Host models. The loss is independent of the model it is attached to. We attach it to DKT, to AKT (a common attention-based tracer), and to QIKT. QIKT's IRT-style output $\sigma(\alpha+\beta+\zeta)$, with additive terms for knowledge acquisition, knowledge state, and problem solving, already gives interpretable parameters. Adding our loss shows whether containment consistency adds anything beyond per-question interpretability.
Baselines
The naive reference point is each host model trained on the prediction loss alone. It is run on identical splits, with identical sequence truncation and the same hyperparameter search budget.
The strongest existing accuracy methods on ASSISTments 2009 in this file are L-HAKT, which reports AUC 80.22% and ACC 75.10% (L-HAKT), and QDCKT at AUC 0.7893 (QDCKT). MMAKT reports 0.7644, with DKT at 0.7364 under its preprocessing. The original DKT paper reports 0.86, so preprocessing clearly differs across papers. For that reason we re-run every baseline in one shared pipeline rather than compare against quoted numbers.
QDCKT is the most direct competitor because it is the only existing method that enforces an ordering on predictions. We run it with its published constraint and give it the same tuning budget. Q-MCKT is included as the contrastive method designed for sparse questions. L-HAKT is reported where we can reproduce it; otherwise its published number is quoted with the preprocessing caveat.
Beating these baselines means two things together. The violation rate (defined below) must fall significantly. And AUC must stay within 0.3 points or improve on full data, while clearly improving in the low-data regimes.
The key ablations each isolate one ingredient. A fixed-margin variant sets $m$ to a tuned constant and isolates the dynamic part. A variant with no confidence weight sets $w \equiv 1$ and isolates the protection against the feedback loop. A variant that uses only answered pairs isolates the unlabeled-pair supervision.
The structural baselines are a hard conjunctive output head and KT². The hard head computes success as a product of per-skill mastery with guess and slip terms, like the classical DINA model, on top of the same encoder. KT² is run in the 50-student regime.
Why not just…?
Why not just hard-code the AND gate in the output layer? This is the real threat. If a product-of-mastery head gives the same accuracy and zero violations, our margin is redundant, and we will say so. Our reason to expect it to fall short is that Pardos et al. established conjunction for middle-school math items in a static network. In domains like EdNet's English items, partial compensation is plausible, and a hard product cannot represent it. It also caps the flexibility of attention-based encoders. The experiment that settles this compares the hard head, the soft margin, and both combined, reported per dataset. If the hard head wins on math and loses on EdNet, the conclusion is that the soft form is needed where composition is only approximately conjunctive.
Why not just extend QDCKT's fixed penalty to nested skill sets? This is the fixed-margin ablation. The expected failure follows from the AND-gate evidence. For a student who has mastered the extra skills, the correct gap is near zero, and a constant margin forces a spurious drop. For a student who clearly lacks them, the correct gap is large, and a constant margin tuned for the average student is too small. We measure calibration within groups of students sorted by their estimated mastery of the extra skills. The dynamic margin should reduce expected calibration error in both extreme groups, while the fixed margin should be worse in at least one.
Why not just use KT² or classical Bayesian tracing in small classrooms? In the 50-student regime this may well be right. KT² reports outperforming AKT, SAINT and LLMs there. No numeric result was found on Lacuna for KT² on XES3G5M or MOOCRadar, so we re-run it. Our claim is modest: the consistency loss should close most of the gap between neural tracers and KT² at 50 students, and keep neural accuracy at scale. If KT² still wins clearly at 50 students, we report that as the regime where tree-structured probabilistic models should be preferred.
Why not just apply SHAP to an unconstrained model? Additive attributions cannot express the effect that one missing prerequisite collapses success. They also only describe a model that may itself contradict the ordering, as QDCKT documents. We test this with the intervention check described below rather than argue it.
Proposed Experiments
Datasets. We use ASSISTments 2009 and Algebra05 because both have many multi-skill items. We use the ASSISTments 2005–06 multi-skill subset (184 single-skill, 62 double-skill and 16 triple-skill items) for direct comparison with the Pardos composition estimates. We use XES3G5M and MOOCRadar for the low-resource classroom protocol of KT², and EdNet as a non-math domain where the AND gate may not hold.
Metrics. Accuracy is measured with AUC and RMSE. Calibration is measured with expected calibration error; ReliCD reports it falling from 6.97% to 1.69% on its diagnosis task, which we cite only as a reference scale. Structural faithfulness is measured with the violation rate: the fraction of (student, step, nested pair) triples where the superset item is predicted more likely to be answered correctly than the subset item.
Label-efficiency test. We train on 1%, 5%, 20% and 100% of students, plus the KT² protocol of 50 students with 5 exercises each. The prediction is that the AUC gain from the consistency loss over the naive host model grows monotonically as the labeled fraction shrinks. The variant restricted to answered pairs should lose most of that low-data gain.
Intervention test. For held-out students, we insert a synthetic incorrect response on a skill $k$ and measure the change in predicted success on items requiring $k$ plus other skills, relative to items not requiring $k$. A faithful conjunctive model should show a drop that is concentrated on items requiring $k$ and larger for students with high mastery of the co-required skills.
Composition recovery. On the ASSISTments 2005–06 subset, we estimate effective knowledge with one of two skills mastered from each model's predictions and compare it with the ≈0.05 reported by Pardos et al.
Decisive outcome. The proposal succeeds if three things hold together: the dynamic margin beats both the fixed margin and the naive model on violation rate and low-data AUC, it matches or beats the hard conjunctive head on EdNet, and it stays within noise of the strongest full-data baseline. If the hard head matches it everywhere, the soft margin is unnecessary and the recommendation becomes the simpler architecture.
References
- The Composition Effect: Conjuntive or Compensatory? An Analysis of Multi-Skill Math Questions in ITS
- Question Difficulty Consistent Knowledge Tracing
- Deep Metric Learning with Hierarchical Triplet Loss
- CNN-Based Patch Matching for Optical Flow with Thresholded Hinge Embedding Loss
- A Question-centric Multi-experts Contrastive Learning Framework for Improving the Accuracy and Interpretability of Deep Sequential Knowledge Tracing Models
- Difficulty-Focused Contrastive Learning for Knowledge Tracing with a Large Language Model-Based Difficulty Prediction
- Towards LLM-Empowered Knowledge Tracing via LLM-Student Hierarchical Behavior Alignment in Hyperbolic Space
- ReliCD: A Reliable Cognitive Diagnosis Framework with Confidence Awareness
- A Hierarchical Probabilistic Framework for Incremental Knowledge Tracing in Classroom Settings
- Deep Knowledge Tracing
- Improving Interpretability of Deep Sequential Knowledge Tracing Models with Question-centric Cognitive Representations
- Multi-factor Memory Attentive Model for Knowledge Tracing