Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository

Radhika Kapoor, Sang T. Truong, Nick Haber, Maria Araceli Ruiz-Primo, Benjamin W. Domingue

TL;DR

This paper introduces a method for automatically predicting the difficulty of reading comprehension test items using text content and student metadata. By mapping raw student pass rates across grades 3–8 to a unified psychometric scale, the authors demonstrate that standard Ridge regression paired with either traditional linguistic features or Large Language Model (LLM) embeddings can predict item difficulty with high accuracy (correlation of 0.77). This approach could significantly reduce the time and financial costs associated with pre-testing educational assessment items.

What Problem Does This Paper Address?

Creating standardized tests is an expensive, time-consuming process. Typically, human experts draft questions, which must then be "field-tested" on thousands of students to determine their actual difficulty before they can be used in real exams.

Predicting this difficulty automatically from text is challenging due to the "p-value problem." In psychometrics, an item's p-value is the percentage of students who answer it correctly. However, a p-value of 0.60 for a 3rd-grade reading item means something entirely different than a 0.60 for an 8th-grade item, because 8th graders have significantly higher baseline reading abilities. To build an effective predictive model, researchers must first figure out how to normalize these scores across different grade levels onto a single, continuous scale, and then determine which textual features actually drive reading complexity.

Methods

The authors tackle the problem using a two-stage pipeline:

  1. Psychometric Rescaling (Vertical Equating): To make difficulties comparable across grades, the authors map raw p-values to a common vertical logit scale. They leverage a 1-Parameter Logistic (1PL) Item Response Theory (IRT) Rasch model combined with publicly available national student achievement norms (NWEA MAP 2020). This converts the target variable into a true measure of "item easiness" ($b_j$).
  2. Feature Extraction & Prediction: They extract four categories of predictors from the text and context:
  3. Context Features: State, grade level, and year.
  4. Test Features: Formatting cues (e.g., whether text is bolded, italicized, or if a question includes a direct excerpt).
  5. Linguistic Features: Over 100 metrics generated via Coh-Metrix, spanning descriptive statistics, readability scores (like Flesch-Kincaid), text cohesion, syntactic complexity, and word complexity.
  6. LLM Embeddings: Dense text representations generated by merging the passage, question, and answers into a single string. They test three models: BERT-base, ModernBERT, and LLaMA-3.1-8B. The resulting high-dimensional embeddings are compressed using Principal Component Analysis (PCA) to retain 80% of the variance.

Finally, they train a Ridge ($L_2$) penalized regression model using 5-fold cross-validation to predict the rescaled item difficulty.

Datasets

The study evaluates its methods on a newly compiled repository of standardized reading comprehension tests:

  • Source: New York State Testing Program (NYSTP) and Texas STAAR exams.
  • Scope: Grades 3–8, spanning the years 2018–2023.
  • Size: 1,076 multiple-choice items based on 170 unique reading passages.

Key Insights and Ideas

  • LLMs vs. Hand-Crafted Features: One of the most striking findings is that models using only LLM embeddings performed almost identically to models using only hand-crafted linguistic features (Coh-Metrix). This suggests that modern transformers implicitly capture complex text properties—like syntactic density and referential cohesion—without requiring specialized psycholinguistic software.
  • Redundancy in Combination: Combining LLM embeddings with traditional linguistic features yielded only marginal improvements. For practitioners, this means you likely only need one or the other, allowing teams to choose between the human interpretability of linguistic metrics or the computational simplicity of raw embeddings.
  • ModernBERT Outperforms: Among the embedding models, ModernBERT achieved the best standalone performance, validating its architecture for encoder-based sequence tasks in specialized domains.

Key Results

The performance of the models was evaluated using Root Mean Squared Error (RMSE) and Pearson Correlation Coefficient between true and predicted difficulty:

  • Baseline Model (Mean Predictor): RMSE = 0.92
  • Best Overall Model (All Features + ModernBERT/LLaMA PCA): RMSE = 0.59, Correlation = 0.77
  • Linguistic Features Only: RMSE = 0.62, Correlation = 0.75
  • ModernBERT Embeddings Only: RMSE = 0.62, Correlation = 0.76

Crucially, the models could predict item difficulty reliably without including context characteristics (like what grade the test was for), meaning the text content alone carries the vast majority of the signal needed to determine difficulty.

Limitations and Future Work

While promising, the authors note several limitations:

  • Geographic Constraint: The dataset is limited to public school data from New York and Texas, which may restrict how well the findings generalize to other states or international student populations.
  • Lack of Domain Tuning: The LLMs used were pre-trained on general web text. Fine-tuning these models on K-12 specific educational corpora could unlock better performance.
  • Aggregated Data: The model predicts average item difficulty for a population, which inherently oversimplifies individual student differences and unique test-taking contexts.

Future work will focus on expanding the dataset to international benchmarks like PISA and integrating these difficulty models directly into automated item generation pipelines to give test creators real-time feedback.

Summary updated

Abstract

Prediction of item difficulty based on its text content is of substantial interest. In this paper, we focus on the related problem of recovering IRT-based difficulty when the data originally reported item p-value (percent correct responses). We model this item difficulty using a repository of reading passages and student data from US standardized tests from New York and Texas for grades 3-8 spanning the years 2018-23. This repository is annotated with meta-data on (1) linguistic features of the reading items, (2) test features of the passage, and (3) context features. A penalized regression prediction model with all these features can predict item difficulty with RMSE 0.59 compared to baseline RMSE of 0.92, and with a correlation of 0.77 between true and predicted difficulty. We supplement these features with embeddings from LLMs (ModernBERT, BERT, and LlAMA), which marginally improve item difficulty prediction. When models use only item linguistic features or LLM embeddings, prediction performance is similar, which suggests that only one of these feature categories may be required. This item difficulty prediction model can be used to filter and categorize reading items and will be made publicly available for use by other stakeholders.
BibTeX
@inproceedings{kapoor2025prediction,
  author = {Radhika Kapoor and Sang T. Truong and Nick Haber and Maria Araceli Ruiz-Primo and Benjamin W. Domingue},
  title = {{Prediction of Item Difficulty for Reading Comprehension Items by Creation of Annotated Item Repository}},
  booktitle = {AI4CHL @ ICLR},
  year = {2025},
  url = {https://openreview.net/forum?id=Cv7w713EOI},
}
Figure