TL;DR
This paper introduces a new workflow component for the Additive Factors Model (AFM) that explicitly accounts for "slipping" – when students incorrectly apply a skill they've learned. The study shows that this enhanced model (AFM+S) generally provides a better fit to student learning data and leads to more accurate parameter estimates, particularly for learning rates. This suggests that traditional AFM might underestimate learning when students exhibit slipping behavior.
What Problem Does This Paper Address?
Many educational data mining models, like the widely used Additive Factors Model (AFM), aim to understand how students learn. AFM estimates a student's prior knowledge, the difficulty of skills, and how quickly they learn. However, AFM has a limitation: it doesn't explicitly model "slipping." Slipping occurs when a student knows a skill but still makes mistakes, perhaps due to inattention or a misunderstanding. Without accounting for this, AFM assumes that with enough practice, a student's mastery of a skill will approach 100%. This can lead to inaccurate parameter estimates, particularly underestimating learning rates, because the model tries to explain persistent errors by assuming slower learning. This paper investigates how incorporating slipping parameters impacts AFM's model fit and the resulting parameter estimates.
Methods
The core contribution is the development of a new workflow component for the LearnSphere platform that implements both the standard AFM and an extended version, AFM+S, which includes explicit parameters for skill slipping. This component is built using Python and leverages bounded logistic regression to handle slipping.
The authors then applied this component to five publicly available datasets from DataShop. They used cross-validation (stratified, unstratified, student-level, and item-level) to compare the model fit of AFM and AFM+S. Following the model fit assessment, they compared the parameter estimates (skill intercepts, skill slopes/learning rates, and student intercepts) between the two models using paired Wilcoxon Signed-Rank tests to identify systematic differences.
Datasets
The study utilized five datasets from the PSLC DataShop:
- Geometry [5]
- Equation Solving [6, 7]
- Number Line Estimation [8]
- Writing 1 [9]
- Writing 2 [10]
These datasets represent various domains of student learning and were chosen to provide a diverse evaluation of the AFM+S model.
Key Insights and Ideas
- Slipping Matters: The most significant insight is that explicitly modeling slipping (AFM+S) generally leads to a better fit of student learning data compared to the standard AFM, especially in datasets where slipping is present.
- Underestimated Learning: When slipping occurs, AFM tends to underestimate learning rates. The AFM+S model, by accounting for slipping, provides higher estimates for skill learning rates.
- Parameter Compensation: The standard AFM appears to compensate for the effects of slipping by adjusting other parameters. Specifically, it estimates higher student intercepts, suggesting it attributes more of the error to general student factors rather than skill-specific slipping.
- Model Choice is Crucial: The findings highlight that researchers using AFM should be cautious when interpreting parameter estimates, particularly learning rates, in contexts where slipping might be occurring. Transitioning to AFM+S could offer a more accurate representation of student learning.
Key Results
- Improved Model Fit: AFM+S demonstrated a significantly better fit to the data across the five datasets and four cross-validation types (V=1350.5, p < 0.01). This improvement was particularly pronounced in unstratified and stratified cross-validation.
- Dataset-Specific Differences: The benefits of AFM+S were most evident in the Geometry and Equation Solving datasets, where slipping rates were more substantial. For datasets with near-zero slipping rates (Number Line, Writing 1 & 2), AFM and AFM+S performed nearly identically.
- Parameter Estimate Differences:
- AFM+S estimated higher skill intercepts (V=257.5, p < 0.01) and learning rates (slopes) (V=117, p < 0.01) compared to AFM.
- Conversely, AFM estimated higher student intercepts (V=9226, p < 0.01) than AFM+S.
Limitations and Future Work
- Statistical Power: The authors acknowledge that for student and item cross-validation, only one run of 2-fold cross-validation was performed due to concerns about Type I error. This might have limited statistical power to detect differences in model fit for these specific cross-validation types.
- Dataset Specificity: While the results are promising, they are based on five specific datasets. Further validation on a wider range of datasets would strengthen the generalizability of the findings.
- Future Work: The paper showcases the new LearnSphere workflow component, suggesting its utility for researchers who suspect slipping in their data. The authors imply that this component can be used to investigate these phenomena, and the results suggest that researchers should consider using AFM+S when slipping is suspected.
Summary updated