Scalable Learning of Item Response Theory Models

Susanne Frick, Amer Krivosija, Alexander Munteanu

Abstract

Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data that indicates the quality of their corresponding answers. Classical psychometric assessments are based on a relatively small number of examinees and items, say a class of $200$ students solving an exam comprising $10$ problems. More recent global large scale assessments such as PISA, or internet studies, may lead to significantly increased numbers of participants. Additionally, in the context of Machine Learning where algorithms take the role of examinees and data analysis problems take the role of items, both $n$ and $m$ may become very large, challenging the efficiency and scalability of computations. To learn the latent variables in IRT models from large data, we leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets. We develop coresets for their use in alternating IRT training algorithms, facilitating scalable learning from large data.

Introduction

Item Response Theory (IRT) is the foundational framework in psychometrics for estimating latent traits—such as intelligence, health, or proficiency—based on responses to a set of test items. While traditionally applied to small classroom settings, the advent of global assessments like the Programme for International Student Assessment (PISA), which involves over 600,000 participants, and the rise of machine learning benchmarking have pushed IRT into the realm of "big data."

In these modern contexts, the number of examinees ($n$) and items ($m$) can reach hundreds of thousands, making standard alternating optimization procedures computationally prohibitive. These iterative algorithms often exceed the memory limits of standard workstations and require days or weeks to converge. This paper addresses these scalability challenges by introducing a framework based on coresets—small, reweighted subsets of data that provide mathematical guarantees for approximating the full dataset's loss function.

Methodology: IRT Models and Coresets

The research focuses on the two most common IRT variants: the Two-Parameter Logistic (2PL) and Three-Parameter Logistic (3PL) models. The probability of a correct response in these models is defined by the discrimination parameter ($a$), the difficulty parameter ($b$), and, in the case of 3PL, a guessing parameter ($c$).

Item Characteristic Curves for 2PL and 3PL models showing the impact of discrimination (a), difficulty (b), and guessing (c) parameters.

The Logistic Connection

A key insight of this work is the mathematical equivalence between the subproblems of 2PL IRT and logistic regression. By recognizing this connection, the authors adapt sensitivity sampling and leverage score techniques from linear model compression to the IRT domain. A significant technical hurdle in alternating optimization is that the "labels" effectively flip signs between iterations. The authors provide a formal proof (Lemma 3.1) that leverage scores are invariant to these sign flips, allowing a single coreset to be reused throughout the entire iterative training process.

Taming the 3PL Guessing Parameter

The 3PL model introduces a "guessing" parameter ($c$), which makes the objective function non-convex and bounded, presenting a much harder optimization problem. The authors developed a novel coreset construction for 3PL by decomposing the loss into shifted logistic and sigmoid components, proving that the "range space" of these functions has a bounded VC-dimension, which ensures the coreset remains a faithful approximation.

Experimental Evaluation

The proposed framework was evaluated using both synthetic datasets (up to 500,000 examinees) and real-world datasets:

  • SHARE: The Survey of Health, Ageing and Retirement in Europe ($n \approx 139,000$).
  • NEPS: The National Educational Panel Study of German 9th-graders ($n \approx 11,500$).
Relative Error and Mean Absolute Deviation (MAD) as a function of coreset size for the SHARE and NEPS datasets.

As shown in the figure above, the approximation error (both relative error and MAD) drops sharply as the coreset size increases, quickly reaching a plateau where the coreset provides a high-fidelity representation of the full dataset with only a fraction of the original data points.

Key Results and Efficiency Gains

  • Computational Speedup: For 3PL models, the use of coresets reduced total running times by up to 86%. In one large-scale synthetic experiment, a task that would have taken 6.5 days was completed in just 3.8 days.
  • Memory Optimization: For 2PL models, memory usage was reduced to less than 1% of the original dataset size without sacrificing statistical accuracy.
  • Statistical Fidelity: Despite the massive compression, the recovered latent parameters (ability $\theta$, discrimination $a$, and difficulty $b$) remained nearly identical to those calculated from the full dataset, with relative errors typically staying below 0.05.
  • Robustness: The coreset approach consistently outperformed uniform subsampling, particularly on sparse or imbalanced real-world data, where uniform sampling fails to provide multiplicative error guarantees.

Conclusion and Future Work

This work demonstrates that coresets are a powerful tool for scaling IRT models to massive datasets. By leveraging the structural similarities between IRT and logistic regression, the authors provide a framework that offers both theoretical guarantees and practical performance gains.

Future research directions include extending these guarantees to more complex IRT variants, such as polytomous (multi-category) responses and multidimensional IRT. Additionally, while coresets reduce data storage, the latent parameter matrices themselves still scale with $n$ and $m$; future work may explore "sketching" techniques to further compress these parameter spaces.

Summary updated

Abstract

Item Response Theory (IRT) models aim to assess latent abilities of $n$ examinees along with latent difficulty characteristics of $m$ test items from categorical data that indicates the quality of their corresponding answers. Classical psychometric assessments are based on a relatively small number of examinees and items, say a class of $200$ students solving an exam comprising $10$ problems. More recent global large scale assessments such as PISA, or internet studies, may lead to significantly increased numbers of participants. Additionally, in the context of Machine Learning where algorithms take the role of examinees and data analysis problems take the role of items, both $n$ and $m$ may become very large, challenging the efficiency and scalability of computations. To learn the latent variables in IRT models from large data, we leverage the similarity of these models to logistic regression, which can be approximated accurately using small weighted subsets called coresets. We develop coresets for their use in alternating IRT training algorithms, facilitating scalable learning from large data.
BibTeX
@inproceedings{frick2024scalable,
  author = {Susanne Frick and Amer Krivosija and Alexander Munteanu},
  title = {{Scalable Learning of Item Response Theory Models}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=xuvSmn6dXn},
}
Figure