TL;DR
This paper introduces UPSC2M, a massive dataset of two million multiple-choice question attempts from over 46,000 students preparing for a highly competitive Indian exam. They propose and benchmark two core tasks: predicting question difficulty from text alone and predicting student performance based on their past interactions, aiming to advance scalable, personalized educational systems.
What Problem Does This Paper Address?
The dream of truly adaptive learning platforms – systems that tailor educational content and assessments to individual learners in real-time – hinges on two fundamental capabilities: accurately estimating the difficulty of educational material and predicting how well a student will perform on a given question. While existing research has explored these areas, many studies rely on small, domain-specific datasets, limiting their applicability to real-world, high-stakes educational settings. This gap hinders the development of robust, scalable, and personalized learning experiences that can adapt to diverse learners and complex subject matter.
Methods
The paper proposes two benchmark tasks:
- Question Difficulty Estimation: This task aims to predict the empirical difficulty of a multiple-choice question using only its textual content (stem and options). The empirical difficulty is defined as 1 minus the proportion of students who answered the question correctly. The authors use a simple baseline approach involving encoding the question text with a frozen pre-trained text encoder and training a small Multi-Layer Perceptron (MLP) to predict the difficulty score.
- Student Performance Prediction: This task focuses on predicting whether a student will answer a given question correctly, based on their historical interaction data. The input to the models is the student's ID and the question ID, and the output is the predicted probability of a correct answer. The authors evaluate several baselines, including:
- Random and Zero Predictors: Trivial baselines for verification.
- Difficulty-Based Heuristic: Predicts correctness based solely on the question's empirical difficulty.
- Collaborative Filtering (CF) methods: Including SVD, NMF, and bias-only models, which treat the problem as a matrix completion task using observed correctness labels.
Datasets
The primary dataset introduced is UPSC2M.
- It comprises 1,962,573 question attempts from 46,235 students.
- These attempts cover 8,973 unique questions across seven subject areas from the Union Public Service Commission (UPSC) examination.
- Each attempt includes the student ID, question ID, selected answer, correctness, and time taken.
- The dataset is split into training, validation, and test sets for both benchmark tasks.
Key Insights and Ideas
- High-Stakes Data is Crucial: The UPSC exam context is highlighted as particularly valuable due to its competitiveness, broad subject coverage, and the need for higher-order reasoning, making it a challenging yet realistic testbed for educational modeling.
- Textual Features Alone are Limited for Difficulty: While semantic features from question text carry some signal for difficulty estimation, the improvement over a simple mean baseline is modest, suggesting that richer features (like behavioral priors) are needed for more accurate predictions.
- Item Difficulty is a Strong Prior for Student Performance: The difficulty-based heuristic significantly outperforms random and zero predictors, indicating that question difficulty alone is a powerful signal for predicting student success, even without personalization.
- Collaborative Filtering Struggles with Sparsity: Despite using standard CF techniques, the models did not significantly outperform the bias-only model. This is attributed to the high sparsity of the student-question matrix (99.62%), which likely hinders the learning of effective representations and student neighborhoods.
- Response Time is a Missed Signal: The authors acknowledge that response time, a readily available signal in their dataset, was not incorporated into their current models but could offer valuable insights into fluency and hesitation.
Key Results
- Question Difficulty Estimation: The text embedding regression baseline achieved a modest improvement, reducing RMSE by 7.1% and MAE by 9.2% compared to a dummy regressor.
- Student Performance Prediction:
- The Difficulty-Based Heuristic achieved an accuracy of 0.6698.
- Among collaborative filtering methods, the Bias Only CF model performed best with an accuracy of 0.6788, AUC of 0.7210, and Brier score of 0.2051.
- More complex CF models like SVD and NMF did not yield significant gains over the bias-only model.
Limitations and Future Work
- Difficulty Labels: The current difficulty labels are based solely on correctness rates and do not incorporate temporal information or student-specific variations. Future work could involve joint modeling of student and item characteristics, potentially including response times.
- Learner Activity: The high prevalence of low-activity learners (median of 8 questions attempted per student) may limit the generalization capabilities of collaborative filtering models.
- Response Time: Response time features, which could signal fluency or hesitation, were not incorporated into the current models.
- Domain Specificity: While UPSC2M is large and diverse, its focus on a single high-stakes exam context might limit direct transferability to other educational domains.
The authors envision UPSC2M and its benchmark tasks as a strong foundation for building more expressive, interpretable, and personalized models of learner behavior, paving the way for truly adaptive and scalable educational systems.
Summary updated