TL;DR
This paper introduces a method for automatically predicting the difficulty of reading comprehension test items using text content and student metadata. By mapping raw student pass rates across grades 3–8 to a unified psychometric scale, the authors demonstrate that standard Ridge regression paired with either traditional linguistic features or Large Language Model (LLM) embeddings can predict item difficulty with high accuracy (correlation of 0.77). This approach could significantly reduce the time and financial costs associated with pre-testing educational assessment items.
What Problem Does This Paper Address?
Creating standardized tests is an expensive, time-consuming process. Typically, human experts draft questions, which must then be "field-tested" on thousands of students to determine their actual difficulty before they can be used in real exams.
Predicting this difficulty automatically from text is challenging due to the "p-value problem." In psychometrics, an item's p-value is the percentage of students who answer it correctly. However, a p-value of 0.60 for a 3rd-grade reading item means something entirely different than a 0.60 for an 8th-grade item, because 8th graders have significantly higher baseline reading abilities. To build an effective predictive model, researchers must first figure out how to normalize these scores across different grade levels onto a single, continuous scale, and then determine which textual features actually drive reading complexity.
Methods
The authors tackle the problem using a two-stage pipeline:
- Psychometric Rescaling (Vertical Equating): To make difficulties comparable across grades, the authors map raw p-values to a common vertical logit scale. They leverage a 1-Parameter Logistic (1PL) Item Response Theory (IRT) Rasch model combined with publicly available national student achievement norms (NWEA MAP 2020). This converts the target variable into a true measure of "item easiness" ($b_j$).
- Feature Extraction & Prediction: They extract four categories of predictors from the text and context:
- Context Features: State, grade level, and year.
- Test Features: Formatting cues (e.g., whether text is bolded, italicized, or if a question includes a direct excerpt).
- Linguistic Features: Over 100 metrics generated via Coh-Metrix, spanning descriptive statistics, readability scores (like Flesch-Kincaid), text cohesion, syntactic complexity, and word complexity.
- LLM Embeddings: Dense text representations generated by merging the passage, question, and answers into a single string. They test three models: BERT-base, ModernBERT, and LLaMA-3.1-8B. The resulting high-dimensional embeddings are compressed using Principal Component Analysis (PCA) to retain 80% of the variance.
Finally, they train a Ridge ($L_2$) penalized regression model using 5-fold cross-validation to predict the rescaled item difficulty.
Datasets
The study evaluates its methods on a newly compiled repository of standardized reading comprehension tests:
- Source: New York State Testing Program (NYSTP) and Texas STAAR exams.
- Scope: Grades 3–8, spanning the years 2018–2023.
- Size: 1,076 multiple-choice items based on 170 unique reading passages.
Key Insights and Ideas
- LLMs vs. Hand-Crafted Features: One of the most striking findings is that models using only LLM embeddings performed almost identically to models using only hand-crafted linguistic features (Coh-Metrix). This suggests that modern transformers implicitly capture complex text properties—like syntactic density and referential cohesion—without requiring specialized psycholinguistic software.
- Redundancy in Combination: Combining LLM embeddings with traditional linguistic features yielded only marginal improvements. For practitioners, this means you likely only need one or the other, allowing teams to choose between the human interpretability of linguistic metrics or the computational simplicity of raw embeddings.
- ModernBERT Outperforms: Among the embedding models, ModernBERT achieved the best standalone performance, validating its architecture for encoder-based sequence tasks in specialized domains.
Key Results
The performance of the models was evaluated using Root Mean Squared Error (RMSE) and Pearson Correlation Coefficient between true and predicted difficulty:
- Baseline Model (Mean Predictor): RMSE = 0.92
- Best Overall Model (All Features + ModernBERT/LLaMA PCA): RMSE = 0.59, Correlation = 0.77
- Linguistic Features Only: RMSE = 0.62, Correlation = 0.75
- ModernBERT Embeddings Only: RMSE = 0.62, Correlation = 0.76
Crucially, the models could predict item difficulty reliably without including context characteristics (like what grade the test was for), meaning the text content alone carries the vast majority of the signal needed to determine difficulty.
Limitations and Future Work
While promising, the authors note several limitations:
- Geographic Constraint: The dataset is limited to public school data from New York and Texas, which may restrict how well the findings generalize to other states or international student populations.
- Lack of Domain Tuning: The LLMs used were pre-trained on general web text. Fine-tuning these models on K-12 specific educational corpora could unlock better performance.
- Aggregated Data: The model predicts average item difficulty for a population, which inherently oversimplifies individual student differences and unique test-taking contexts.
Future work will focus on expanding the dataset to international benchmarks like PISA and integrating these difficulty models directly into automated item generation pipelines to give test creators real-time feedback.
Summary updated