No Task Left Behind: Multi-Task Learning of Knowledge Tracing and Option Tracing for Better Student Assessment

Suyeong An, Junghoon Kim, Minsam Kim, Juneyoung ParkAAAI 2022 · DOI

Introduction

In AI-driven education (AIEd), accurately assessing a student's knowledge state is critical for effective personalization. Traditional Knowledge Tracing (KT) models typically employ a dichotomous approach, focusing solely on whether a student's response is correct or incorrect. This binary classification overlooks significant diagnostic information contained within the specific incorrect options (distractors) a student selects. Conversely, Option Tracing (OT) predicts the specific choice made by a student but often fails to leverage the explicit correctness signal, potentially tracking behavioral patterns rather than actual knowledge.

To address these limitations, researchers have developed DP-MTL (Dichotomous-Polytomous Multi-Task Learning), a framework that integrates both KT and OT objectives to achieve a more granular and accurate assessment of student learning.

Methodology: The DP-MTL Framework

The core of DP-MTL is a multi-task objective function that balances dichotomous correctness prediction with polytomous option choice prediction. The combined loss function is defined as:

$$L_{DP} = \lambda L_D + (1 - \lambda) L_P$$

where $L_D$ represents the standard KT loss and $L_P$ represents the OT loss. The hyperparameter $\lambda \in [0, 1]$ allows the model to balance these two objectives.

KT as a Regularization Mechanism

A key theoretical contribution of this work is the demonstration that the KT objective acts as a regularizer for the OT task. In instances where a student provides an incorrect response, the KT loss encourages the model to reduce the probability of the correct option while distributing the remaining probability across all distractors. This prevents the model from over-fitting to a single incorrect choice and leads to more stable student representations.

Comparison of probability updates across Knowledge Tracing (KT), Option Tracing (OT), and the proposed DP-MTL framework.

Position-Agnostic Architecture

To avoid "positional bias"—where models learn a student's tendency to pick a specific letter (e.g., 'C')—DP-MTL utilizes option embeddings. This architecture allows the model to remain agnostic to the number and order of choices, focusing instead on the semantic content of the options. Furthermore, the framework is model-agnostic and can be integrated into various architectures such as Item Response Theory (IRT), Neural Matrix Factorization (NMF), and Bidirectional LSTMs (BiDKT).

Experimental Evaluation

The framework was evaluated on two datasets: ENEM (a dense dataset from Brazil's national college entrance exam) and TOEIC (a sparse dataset from an online tutoring system).

Key Results

The integration of dichotomous and polytomous tasks yielded significant improvements across multiple metrics:

  1. Score Prediction (SP): DP-MTL significantly outperformed baseline models in predicting final exam scores. On the TOEIC dataset, the Mean Absolute Error (MAE) was reduced by up to 24.9% compared to standard KT models.
  2. Knowledge Tracing Performance: Even when the primary goal is binary correctness prediction (AUC), the inclusion of the option-tracing task provided a performance boost, particularly in sparse data environments.
  3. Optimal Task Balancing: Empirical results consistently showed that the best performance is achieved when $\lambda$ is between 0 and 1, typically in the 0.5 to 0.7 range. This confirms that the two tasks are complementary and that multi-task learning is superior to focusing on either task in isolation.
Average performance rank across different models as a function of the task-weighting hyperparameter $\lambda$.

Conclusion

DP-MTL demonstrates that "no task should be left behind" in student assessment. By simultaneously learning from correctness and specific option choices, the model builds a holistic representation of student knowledge. This granular understanding not only improves performance on standard benchmarks but also provides a foundation for more effective personalized content recommendation and diagnostic feedback in digital learning environments.

Summary updated

BibTeX
@inproceedings{an2022no,
  author = {Suyeong An and Junghoon Kim and Minsam Kim and Juneyoung Park},
  title = {{No Task Left Behind: Multi-Task Learning of Knowledge Tracing and Option Tracing for Better Student Assessment}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i4.20364},
}
Figure