When a student gets a word problem wrong, a tutoring system usually cannot tell whether the reading or the decimal arithmetic failed. Models that track student knowledge over time (knowledge tracing) respond in one of three ways. They blame every skill the question requires equally, blame the weakest one, or average per-skill predictions. Each choice puts mastery estimates in the wrong place, and those estimates decide what the student practices next. The information needed to fix this is often already logged. On a multiple-choice item, the specific wrong option a student picks is not random. A distractor such as "0.5 instead of 0.05" is what a student produces after reading the problem correctly and then misplacing a decimal. That choice points at one skill and clears the other.
We propose to treat option choice as a second prediction target, trained jointly with correctness, and to connect the two through an explicit latent variable: which of the item's required skills the student has actually mastered. Correctness follows an AND-gate, so failing any required skill makes failure likely. Each distractor is modeled as the typical product of a particular pattern of skill failures. The joint objective is the multi-task part. The posterior over failure patterns after the student answers is the credit assignment. The main claim is that, compared with correctness-only multi-skill models, this joint training makes per-skill updates both more accurate and more identifiable. The test is whether the model better predicts later performance on single-skill items, where credit assignment errors cannot hide.
Core Claim and Positioning
Two lines of prior work each have half of this. No Task Left Behind (DP-MTL) trains correctness and option prediction jointly with a fixed weight $\lambda$. It shows that the correctness loss regularizes distractor prediction and reports TOEIC score-prediction MAE reduced by up to 24.9%. However, it treats each item as one undifferentiated unit, so it cannot say which subskill produced a distractor. EAKT keeps an explicit per-skill knowledge state, linked to items through a continuous skill-requirement matrix (Q-matrix) obtained by factorization. It uses only binary correctness, and its skill-to-item link is fixed before training. We combine the two halves through a discrete latent failure pattern, which neither paper has.
Psychometric work supplies the composition rule and limits the design. The Composition Effect found that multi-skill ASSISTments math items behave conjunctively. DAS3H shows that compensatory, averaging-style models falsely credit a weak skill when a strong co-occurring skill carried the answer. The gating mechanics of LR-DBN (Logistic Regression in a Dynamic Bayes Net Models Multiple Subskills Better!) also showed that fitting subskills jointly beats tracing them independently (Comparison of methods to trace multiple subskills). We keep the conjunctive structure. Our addition is a second, polytomous observation channel (one with more than two possible outcomes) that breaks the symmetry a single binary outcome leaves among the skills.
The research file also suggests a more obvious intersection: applying gradient surgery such as PCGrad or RotoGrad to per-skill gradients. We think that idea is weak. It fixes an optimization symptom, but the real problem is that one bit of feedback per item cannot identify a failure among several skills, and reshaping gradients does not add information. A conjunctive likelihood already sends most of the failure gradient to the least-mastered skill. We keep gradient surgery only as a secondary comparison on the correctness/option task pair.
Model
For student $u$ at time $t$, a sequence encoder outputs mastery probabilities $m_{k,t}$ for each skill $k$. Item $q$ requires the skill set $S_q$, which usually has at most four elements. A latent pattern $z \in \{0,1\}^{|S_q|}$ records which required skills the student has actually mastered. Under the independence assumption, $P(z) = \prod_{k} m_k^{z_k}(1-m_k)^{1-z_k}$. The observed option $r$ then has likelihood
$$ P(r \mid q) = \sum_{z} P(z)\,\Big[\mathbb{1}[z=\mathbf{1}]\,\big((1-s_q)\mathbb{1}[r=r^*] + s_q\,\pi_q(r\mid \text{slip})\big) + \mathbb{1}[z\neq\mathbf{1}]\,\big(g_q\mathbb{1}[r=r^*] + (1-g_q)\,\pi_q(r \mid z)\big)\Big], $$
Here $r^*$ is the correct option, $s_q$ and $g_q$ are slip and guess rates, and $\pi_q(r\mid z)$ is the distribution over wrong options given a failure pattern. Because there are at most 16 patterns, the sum is exact. To stop $\pi_q$ from absorbing everything, we parameterize it with a sparse distractor signature: each wrong option gets a learned, position-agnostic embedding scored against embeddings of the failed skills. This reuses DP-MTL's idea of removing option-position bias.
The training loss is $\lambda\,\mathcal{L}_{\text{correct}} + (1-\lambda)\,\mathcal{L}_{\text{option}}$. When $\lambda = 1$, the model reduces to a conjunctive correctness-only tracer. Credit assignment is the posterior $P(z \mid r)$. The mastery state updates using the posterior failure probability of each skill, rather than the raw outcome.
Research Program
We will build the model on a standard attention-based tracer backbone. Following simpleKT, the backbone uses question-difficulty offsets so that any gain cannot come from a weak encoder. All evaluation will use the question-level, all-in-one protocol of pyKT. pyKT showed that expanding multi-skill items into consecutive single-skill steps can inflate AUC by up to 13%, so expanded protocols are off limits. Every model predicts the whole item from history before that item.
The work has three stages. The first is a controlled simulator. It extends the ATC-style synthetic data used by EAKT with distractors generated from known skill-failure patterns, and it varies how diagnostic the distractors are, from fully informative to random. Here true mastery is known, so credit assignment can be scored directly. The second stage uses real logs with option-level responses: ENEM and TOEIC, as used by DP-MTL. We will also use NIPS34 if preprocessing confirms option-level responses and multiple skill tags per item. The third stage is an analysis of when the auxiliary channel helps. It relates gains to two measured quantities: the fraction of multi-skill items, and how concentrated distractor choices are given performance on other items.
Baselines
The naive reference point is pyKT's Late Fusion Average. This method trains a single-skill-indexed tracer and averages per-skill predicted probabilities for multi-skill items. pyKT found it the most stable of the aggregation heuristics it tested, so it is what a practitioner would deploy today. The strongest predictive baseline is simpleKT, reported at 57 wins, 3 ties and 16 losses against 12 deep tracing models across seven datasets. We include AKT as well, since pyKT found it ahead of second-best models by roughly 1.5–3% AUC and simpleKT reports AKT keeping a usually <0.5% AUC edge on some datasets. For a multi-skill baseline we use EAKT, which reports 84.6% AUC on ASSISTments 2009 against 81.4% for DKT. That number was obtained outside pyKT's leakage-free protocol, so we will rerun EAKT rather than quote it. No reported result was found on Lacuna for absolute AUC of DP-MTL or EAKT on ENEM.
The closest competitor on our specific question is DP-MTL itself, with an IRT-family and a BiDKT backbone and $\lambda$ tuned over its reported 0.5–0.7 range. It uses the same option signal but has no skill structure. The key ablation is our own model at $\lambda=1$: conjunctive and multi-skill, but trained on correctness only. This isolates whether distractors, rather than the AND-gate, produce the gain.
All methods share the same splits, the same item–skill tags, the same hyperparameter search budget (an equal number of trials per method), and matched parameter counts within 10%. Beating the baselines means meeting all three of the following. First, item-level AUC is no worse than simpleKT or AKT. Second, the model shows a statistically significant gain over both DP-MTL and the $\lambda=1$ ablation on the single-skill-followup metric defined below. Third, in simulation, mastery recovery correlates better with ground truth.
Why not just…?
Why not just use a conjunctive output layer on binary data? This is the strongest threat, and it is exactly our $\lambda=1$ ablation. A noisy-AND already routes blame toward the weakest skill. If distractors add nothing beyond that, the simulator will show it: as distractor informativeness drops to random, our model should converge to the ablation. On real data, any remaining gap on single-skill followups is the contribution. If there is no gap, we will report that the AND-gate alone is sufficient.
Why not just average per-skill predictions, as pyKT recommends? Averaging is a compensatory model. Under the conjunctive behavior documented in The Composition Effect, it should overestimate success when one skill is missing and wrongly credit weak skills, as DAS3H describes. It may still match us on item-level AUC, because averaging can be well calibrated on aggregate. That is why our decisive metric is per-skill prediction on later items, not item AUC.
Why not just feed option choices into DP-MTL or simpleKT as input features? A flexible encoder given option IDs might learn skill-specific signals on its own. DP-MTL is the version of this that is already tested, and we run it with tuned $\lambda$. We also add a simpleKT variant whose input includes the chosen option embedding. If either matches us on single-skill followups, the latent failure pattern is unnecessary and we will say so. What remains unique to our model is an interpretable posterior over which skill failed, and we score that only in simulation.
Why not just apply gradient surgery across skills? PCGrad's gains appear in settings with genuinely separate task losses, such as Meta-World MT50, where success went from under 40% to about 70%. Per-skill knowledge tracing has one label per item, so there are no separate per-skill losses to project between. We test PCGrad on the two real losses, correctness and option, as a variant of our model, expecting little benefit.
Proposed Experiments and Decisive Checks
The decisive real-data check is single-skill followup AUC. For every multi-skill item a student answered, we take the next single-skill item that involves one of its skills and measure prediction AUC on that item. Correct credit assignment should raise this score, while item-level AUC barely moves. We report the score separately for followups after wrong answers, where the distractor signal exists, and after right answers, where it does not and no gain is expected. That split works as a built-in placebo test.
In simulation, we measure rank correlation between estimated and true per-skill mastery, and the accuracy of the posterior failure pattern against ground truth, across the informativeness sweep. A dose–response relationship, where gains grow with distractor informativeness and vanish at random, would be strong evidence for the mechanism. A flat curve would refute it.
As a secondary analysis, we inspect learned distractor signatures on ENEM for items whose distractors can be traced to a plausible single-skill error, and report how often the top-weighted skill matches. All results are reported as averages over five seeds with confidence intervals, under pyKT's leakage-free protocol.
References
- No Task Left Behind: Multi-Task Learning of Knowledge Tracing and Option Tracing for Better Student Assessment
- Embedding cognitive framework with self-attention for interpretable knowledge tracing
- The Composition Effect: Conjuntive or Compensatory? An Analysis of Multi-Skill Math Questions in ITS
- DAS3H: Modeling Student Learning and Forgetting for Optimally Scheduling Distributed Practice of Skills
- Logistic Regression in a Dynamic Bayes Net Models Multiple Subskills Better!
- Comparison of methods to trace multiple subskills: Is LR-DBN best?
- Gradient Surgery for Multi-Task Learning
- RotoGrad: Gradient Homogenization in Multi-Task Learning
- simpleKT: A Simple But Tough-to-Beat Baseline for Knowledge Tracing
- pyKT: A Python Library to Benchmark Deep Learning based Knowledge Tracing Models