Introduction
In the field of Learning Analytics (LA), researchers frequently compare predictive models to determine which best identifies student outcomes, such as dropout risk. A common practice is to run k-fold cross-validation, calculate average performance metrics (e.g., Accuracy or AUC), and declare the model with the higher average as the superior choice. However, this approach is often a statistical fallacy. Because cross-validation folds share significant training data, the resulting performance estimates are correlated rather than independent. Applying standard Student’s t-tests to these numbers violates independence assumptions, leading to inflated Type I error rates—falsely claiming a model is better when the difference is merely random noise.
Robust Statistical Methods for Model Comparison
To address the "independence illusion," researchers must utilize corrected statistical tests that account for the overlap in training data. The following table outlines four valid approaches evaluated for the two-model comparison task.
Key Validated Techniques:
- Corrected Repeated k-fold CV (10x10cv): This involves running 10-fold cross-validation ten times and applying a variance correction. It is highly regarded for its statistical power and replicability.
- Sorted Runs Sampling: A newer technique where results from multiple runs are sorted by performance before averaging. This helps satisfy independence assumptions without requiring complex variance corrections.
- Corrected Resampled t-test: A modification that adjusts the variance estimate to account for the correlation between resamples.
- 5x2cv Paired t-test: While statistically sound, this method often suffers from lower power compared to the 10x10cv approach.
The Sorted Runs Sampling Scheme
The sorted runs approach is particularly noteworthy for its simplicity and effectiveness in educational data. By sorting the performance of folds across different runs, researchers can better manage the variance inherent in random data splits.
Case Study: Financial Literacy MOOC
The authors applied these methods to a real-world dataset from a Financial Literacy MOOC on Coursera, consisting of 23,537 student observations. The task was to predict third-week activity based on features from the first two weeks. The data was highly unbalanced (1:5 dropout-to-retention ratio), a common characteristic in educational settings.
Results: Accuracy vs. AUC
The study compared a Multi-Layer Perceptron (MLP) against a Classification Tree (CART). While the MLP showed a slightly higher raw Accuracy (0.8269 vs. 0.8265), the corrected 10x10cv p-value was 0.089, indicating the difference was not statistically significant. However, when evaluating AUC, the MLP (0.596) significantly outperformed the Tree (0.566) with a p-value of 0.016. This highlights that raw averages can be misleading and that the choice of metric is critical.
Insights and Future Directions
- Pessimistic Bias: Cross-validation uses only a subset of data for training, which can lead to reported performance lower than what the model would achieve on the full dataset.
- Unbalanced Data: Interestingly, the "Sorted Runs" method becomes more replicable as datasets become more unbalanced, making it ideal for student success prediction.
- Limitations: These tests currently focus on two-model comparisons. Comparing multiple models requires additional corrections (e.g., Bonferroni) to avoid multiple comparison traps. Future research may move toward Bayesian approaches to provide intuitive probabilities of model superiority rather than binary p-values.
Summary updated