Scalable Learning of Item Response Theory Models

Susanne Frick, Amer Krivosija, Alexander Munteanu

Abstract

Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data that indicates the quality of their corresponding answers. Classical psychometric assessments are based on a relatively small number of examinees and items, say a class of $200$ students solving an exam comprising $10$ problems. More recent global large scale assessments such as PISA, or internet studies, may lead to significantly increased numbers of participants. Additionally, in the context of Machine Learning where algorithms take the role of examinees and data analysis problems take the role of items, both $n$ and $m$ may become very large, challenging the efficiency and scalability of computations. To learn the latent variables in IRT models from large data, we leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets. We develop coresets for their use in alternating IRT training algorithms, facilitating scalable learning from large data.

Introduction

Item Response Theory (IRT) is the gold standard in psychometrics for measuring latent traits—such as a student's mathematical ability or an AI agent's reasoning skills—based on their performance on a set of items. As we move toward global-scale assessments or benchmarking massive AI models, standard alternating optimization algorithms for IRT encounter significant scalability hurdles. The computational cost scales with both the number of examinees ($n$) and the number of items ($m$), often leading to datasets that exceed main memory capacity. This paper introduces the first sublinear approximation algorithms for training IRT models using coresets—small, weighted subsets of data that act as accurate proxies for the full dataset.

Item Response Theory Models

The research focuses on two common IRT variants: the Two-Parameter Logistic (2PL) model and the Three-Parameter Logistic (3PL) model. The 2PL model accounts for item difficulty and discrimination, while the 3PL model adds a "guessing" parameter to account for the probability of a correct response by chance.

Item Characteristic Curves (ICCs) for 2PL and 3PL models with varying discrimination ($a$), difficulty ($b$), and guessing ($c$) parameters.

As shown in the figure above, these parameters define the shape of the Item Characteristic Curve (ICC), which represents the probability of a correct response given an examinee's latent ability. The inclusion of the guessing parameter ($c$) in the 3PL model makes the optimization problem non-convex and significantly more challenging than the 2PL case.

Methodology: Coresets for IRT

The core innovation lies in developing coresets specifically for IRT training. A coreset is a small subset of the original data where each point is assigned a weight such that training on this subset yields results nearly identical to the full dataset.

2PL Models and Logistic Regression

The authors demonstrate that the subproblems in 2PL training are mathematically equivalent to logistic regression. They adapt existing logistic coreset techniques, proving that "leverage scores" (importance measures for data points) are invariant to the sign-flips that occur during alternating optimization. This allows a single coreset to remain valid across multiple training iterations.

3PL Models and Non-Convexity

For 3PL models, the authors develop a novel coreset construction by partitioning the loss function into different components. They bound the "sensitivities" of these components using a data-dependent parameter called $\mu$-complexity, ensuring that even the non-convex aspects of the model are accurately represented in the reduced dataset.

Experimental Results

The approach was validated using synthetic data (up to 500,000 examinees) and real-world datasets, including the Survey of Health, Ageing and Retirement in Europe (SHARE) and the National Educational Panel Study (NEPS).

  • Efficiency: The researchers observed training time reductions between 32% and 66% for 2PL models, and up to 86% for 3PL models. In many cases, the coresets required less than 1% of the memory used by the full dataset.
  • Accuracy: The relative error in the objective function was consistently low (often $< 0.05$), and the mean absolute deviation of estimated parameters remained negligible.
  • Scalability: The method successfully reduced the duration of a massive experiment from 6.5 days to just 3.8 days, proving its utility for large-scale psychometric and machine learning applications.

Conclusion and Future Work

This work demonstrates that coresets provide a powerful tool for scaling IRT models to massive datasets without sacrificing statistical rigor. Key insights include the robustness of leverage-score sampling to the iterative nature of IRT training and the effectiveness of partitioning strategies for non-convex loss functions. Future research directions include extending these techniques to multidimensional IRT models, polytomous data (multiple-choice or partial credit), and continuous response models.

Summary updated

Abstract

Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data that indicates the quality of their corresponding answers. Classical psychometric assessments are based on a relatively small number of examinees and items, say a class of $200$ students solving an exam comprising $10$ problems. More recent global large scale assessments such as PISA, or internet studies, may lead to significantly increased numbers of participants. Additionally, in the context of Machine Learning where algorithms take the role of examinees and data analysis problems take the role of items, both $n$ and $m$ may become very large, challenging the efficiency and scalability of computations. To learn the latent variables in IRT models from large data, we leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets. We develop coresets for their use in alternating IRT training algorithms, facilitating scalable learning from large data.
BibTeX
@inproceedings{frick2024scalable,
  author = {Susanne Frick and Amer Krivosija and Alexander Munteanu},
  title = {{Scalable Learning of Item Response Theory Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=xuvSmn6dXn},
}
Figure