Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning

Christopher James MacLellan, Kimberly Stowers, Lisa BradyarXiv 2024 · OpenReview

TL;DR

This paper introduces a method for creating personalized computational models of student learning to predict the effectiveness of different training interventions. By automatically tuning these models to individual students, the researchers demonstrate improved prediction accuracy and enable the generation of counterfactual learning curves, offering a cost-effective way for instructional designers to evaluate interventions before costly human studies.

What Problem Does This Paper Address?

Instructional designers face the significant challenge of determining which training interventions yield the best learning outcomes. Traditional A/B testing, while the gold standard, is expensive and time-consuming, often only comparing a limited number of interventions. Furthermore, A/B tests typically treat interventions as one-size-fits-all, failing to account for individual differences in prior knowledge and learning styles, which can hinder performance gains. The paper aims to address this by leveraging computational models of learning to simulate and predict the impact of various interventions on individual students.

Methods

The core of the paper's methodology lies in using the Apprentice Learner Architecture (ALA) to model student learning. ALA is a framework that incorporates cognitive learning theories to generate theory-driven predictions about human performance. Key components include:

  • Skill-based learning: The model represents knowledge as a set of skills, each with conditions for application, effects on working memory, and a function predicting future reward.
  • Reinforcement Learning: Q-learning is used to update the expected value of skills, allowing the agent to prioritize and select actions.
  • Personalization via HyperOpt: The novel aspect is the automatic tuning of ALA agents to individual students. This is achieved using the HyperOpt toolkit, which iteratively samples skills and cognitive parameters (like disposition towards exploration) to minimize the error between the simulated agent's behavior and the observed human student's behavior on a given task. This process is described as creating "personalized models."
  • Counterfactual Simulation: Once personalized models are created, they are used to simulate student performance under different, hypothetical intervention conditions (e.g., a different problem ordering than what the student actually experienced).

Datasets

The study utilizes data from the publicly available “Fraction Addition and Multiplication” data set accessed via DataShop (Koedinger et al., 2010). This dataset comes from an experiment by Patel et al. (2016) involving 118 sixth graders practicing fraction arithmetic problems in either a blocked or interleaved order. The researchers specifically used "Study 2" data for their analysis.

Key Insights and Ideas

  • Personalization is Key: Generic computational models of learning struggle to accurately predict individual student behavior. Tailoring these models to specific students, by accounting for their unique prior knowledge and cognitive parameters, significantly improves prediction accuracy.
  • Mechanistic Models for Counterfactuals: Unlike abstract "knowledge tracing" models, the ALA's mechanistic approach allows for the generation of plausible counterfactual predictions. This means we can simulate how a student would have performed under an intervention they never actually experienced.
  • Bridging the Gap Between Simulation and Reality: The HyperOpt-driven tuning process effectively bridges the gap between abstract computational models and real-world student performance, demonstrating that these models can learn to emulate individual learning trajectories.
  • Faded Interventions Show Promise: The simulations suggest that a "faded" intervention (starting with blocked practice and gradually transitioning to interleaved) might offer benefits of both blocking (identifying relevant task features) and interleaving (discriminating competing skills), potentially leading to better learning and transfer.

Key Results

  • Reduced Prediction Error: Personalized Apprentice Learner models showed significantly lower prediction error compared to a baseline model (which represented prior work assuming no prior fraction knowledge) when tested on both training and unseen data.
  • Improved Emulation of Human Performance: The personalized models reasonably emulated observed human performance, particularly in addressing the large discrepancy in error rates on the first opportunity that was present in previous, non-personalized models.
  • Counterfactual Predictions Align with Findings: For both high and low-performing students, the personalized models generated understandable counterfactual predictions. These predictions aligned with observed human data for conditions that were part of the original study and offered plausible, though untested, predictions for novel interventions (like the faded condition). For instance, they predicted lower error in blocked vs. interleaved conditions, consistent with prior research.

Limitations and Future Work

  • Evaluating Counterfactuals: A primary limitation is the inherent difficulty in evaluating counterfactual predictions, as there is no ground truth data for interventions that were not administered. The authors acknowledge that future work should design experiments to test these predictions.
  • Testing the Faded Intervention: The paper suggests running a human study to compare the "faded blocked to interleaved" intervention against blocked and interleaved conditions, as their simulations indicate it might yield faster learning and less spiky error rates.
  • Domain Generality: While the approach is demonstrated on fraction arithmetic, future work should explore its application in other domains (psychology, chemistry, math, language learning, engineering) to showcase its generality.
  • Beyond Step-Based Interactions: The current framework is limited to step-based interactions with immediate feedback. Extending it to handle continuous-time interactions and delayed feedback, as found in educational games, is a future direction.
  • Modeling Other Learning Phenomena: The authors suggest expanding the framework to account for phenomena like the "testing effect," where performance improves even without direct feedback, requiring agents to generate internal feedback.
  • Usability for Designers: More effort is needed to make the approach accessible to instructional designers, potentially through user-friendly interfaces that automate the current manual steps.

Summary updated

Abstract

Evaluating different training interventions to determine which produce the best learning outcomes is one of the main challenges faced by instructional designers. Typically, these designers use A/B experiments to evaluate each intervention; however, it is costly and time consuming to run such studies. To address this issue, we explore how computational models of learning might support designers in reasoning causally about alternative interventions within a fractions tutor. We present an approach for automatically tuning models to specific individuals and show that personalized models make better predictions of students' behavior than generic ones. Next, we conduct simulations to generate counterfactual predictions of performance and learning for two students (high and low performing) in different versions of the fractions tutor. Our approach makes predictions that align with previous human findings, as well as testable predictions that might be evaluated with future human experiments.
BibTeX
@misc{maclellan2024evaluating,
  author = {Christopher James MacLellan and Kimberly Stowers and Lisa Brady},
  title = {{Evaluating Alternative Training Interventions Using Personalized Computational Models of Learning}},
  year = {2024},
  eprint = {2408.13684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.13684},
}
Figure