Predictive Student Modeling in Educational Games with Multi-Task Learning

Michael Geden, Andrew Emerson, Jonathan P. Rowe, Roger Azevedo, James C. LesterAAAI 2020 · DOI

Introduction

In the field of game-based learning, the ability to provide personalized support—such as adaptive hints, feedback, or dynamic difficulty adjustment—hinges on accurate predictive student models. These models estimate a student's current knowledge state based on their real-time gameplay behavior. Traditionally, these models have been designed to predict a single aggregate outcome, such as a total post-test score. However, this "single-task" approach often fails to capture the nuance of student learning, as it assumes all test items are equally difficult and map to the same underlying behaviors.

Researchers from North Carolina State University and the University of Central Florida have proposed a shift in this paradigm. By reformulating student performance prediction as a Multi-Task Learning (MTL) problem, they demonstrate that training models to predict answers to individual test questions simultaneously significantly improves predictive accuracy and model robustness.

Methodology

The core innovation of this research is the use of Multi-Task Stacked LSTMs with Residual Connections. Long Short-Term Memory (LSTM) networks are particularly suited for this domain because they can process the sequential nature of gameplay—the specific order and timing of actions a student takes within the game environment.

The Multi-Task Shift

Unlike traditional models that produce a single output (e.g., "the student got 12/17 correct"), the proposed MTL framework features multiple outputs, each corresponding to a specific question on a post-game assessment. This allows the model to learn the specific behavioral patterns associated with different concepts, such as "virus transmission" versus "bacterial structure."

Multi-Task Learning architecture for predicting individual question outcomes.

To optimize this architecture, the researchers implemented several advanced features:

  • Residual Connections: By allowing gradients to skip layers, these connections facilitate more effective training of deep networks.
  • Loss Weighting Strategies: The team compared Unweighted MTL, where all questions are treated with equal importance, against Uncertainty-Weighted MTL. The latter dynamically adjusts the model's focus based on the inherent "noise" or uncertainty in the data for specific tasks.
  • Attention Mechanisms: Self-attention was explored to determine if the model could identify and prioritize critical "learning moments" within the gameplay logs.

Experimental Setup

The study utilized data from CRYSTAL ISLAND, a narrative-driven educational game focused on middle school microbiology. The dataset included logs from 181 students, encompassing a variety of features:

  • Pre-game Surveys: Data on student emotions, interest, and prior knowledge.
  • Trace Logs: Detailed records of gameplay, including 8 action types, 97 action arguments, 24 distinct locations, and time-stamped events.
  • Assessment Labels: A 17-item microbiology content knowledge post-test served as the ground truth for the multi-task outputs.

Results and Discussion

The MTL approach demonstrated a substantial improvement over standard neural network and non-neural baselines. The primary finding was that forcing the model to predict multiple related tasks simultaneously encourages the development of a shared representation of student behavior. This shared representation acts as a powerful form of regularization, preventing the model from overfitting to the noise present in any single aggregate score.

The "More is Better" Trend

A key insight from the study is the relationship between the number of tasks and model performance. As the number of outcome variables (individual questions) increased, the predictive accuracy ($R^2$) of the model improved consistently. This suggests that more granular assessments provide the deep learning model with more "signals" to triangulate the student's true knowledge state.

Impact of the number of outcome variables on model performance ($R^2$).

Key Performance Metrics

  • Predictive Power: The MTL models achieved a peak $R^2$ of 0.51, a significant leap from the 0.41 achieved by single-task sequential models and the 0.37 achieved by the best non-neural baselines (such as Lasso and Random Forest).
  • Error Reduction: The Mean Squared Error (MSE) was reduced from 8.36 in single-task models to 6.92 in the weighted MTL configuration.
  • Weighting and Attention: Interestingly, the researchers found that complex uncertainty weighting and self-attention did not provide significant boosts over simpler MTL configurations. This suggests that for datasets of this scale, the primary benefit comes from the multi-task structure itself rather than additional architectural complexity.

Conclusion and Future Work

This research highlights the potential of Multi-Task Learning to create more accurate and robust student models in educational games. By moving away from aggregate scores, researchers can better capture the multi-faceted nature of learning.

However, limitations remain. The current sample size of 181 students is relatively small for deep learning, and the model currently treats questions as independent tasks. Future work will focus on incorporating the hierarchical nature of knowledge—where understanding one concept is a prerequisite for another—into the model's loss functions. Additionally, improving the interpretability of these "black box" models is crucial for providing actionable insights to educators.

Summary updated

BibTeX
@inproceedings{geden2020predictive,
  author = {Michael Geden and Andrew Emerson and Jonathan P. Rowe and Roger Azevedo and James C. Lester},
  title = {{Predictive Student Modeling in Educational Games with Multi-Task Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i01.5406},
}
Figure