BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing

Aritra Ghosh, Andrew S. Lan

Abstract

Computerized adaptive testing (CAT) refers to a form of tests that are personalized to every student/test taker. CAT methods adaptively select the next most informative question/item for each student given their responses to previous questions, effectively reducing test length. Existing CAT methods use item response theory (IRT) models to relate student ability to their responses to questions and static question selection algorithms designed to reduce the ability estimation error as quickly as possible; therefore, these algorithms cannot improve by learning from large-scale student response data. In this paper, we propose BOBCAT, a Bilevel Optimization-Based framework for CAT to directly learn a data-driven question selection algorithm from training data. BOBCAT is agnostic to the underlying student response model and is computationally efficient during the adaptive testing process. Through extensive experiments on five real-world student response datasets, we show that BOBCAT outperforms existing CAT methods (sometimes significantly) at reducing test length.

Introduction

Computerized Adaptive Testing (CAT) is the core technology behind standardized exams like the GRE or GMAT. Unlike traditional linear tests, CAT systems adaptively select the "best" subsequent question based on a student's previous performance. Historically, these systems have relied on Item Response Theory (IRT), which utilizes static mathematical formulas to determine question informativeness.

However, traditional IRT-based methods face significant limitations. First, they are static; they do not improve by learning from the vast amounts of student response data generated over time. Second, they are rigid, often restricted to simple psychometric models and unable to leverage the representational power of modern deep learning architectures.

The BOBCAT Framework

To address these challenges, we introduce BOBCAT (Bilevel Optimization-Based Computerized Adaptive Testing). BOBCAT re-conceptualizes CAT as a meta-learning problem solved via bilevel optimization.

The BOBCAT framework illustrating the bilevel optimization process.

The framework operates through two nested loops:

  • The Inner Loop (Student Adaptation): For an individual student, the system uses their current responses to "fine-tune" a global response model, creating a student-specific model that estimates their unique knowledge level.
  • The Outer Loop (Meta-Learning): The system evaluates performance on a held-out "meta-set" of questions. It then updates both the global response model and the question selection algorithm to maximize predictive accuracy on this meta-set.

A key technical hurdle in this approach is that question selection is a discrete choice, which typically prevents the use of standard gradient descent. BOBCAT overcomes this by employing a biased approximate estimator based on influence functions. This allows the model to estimate the potential information gain of unselected questions during training, resulting in more stable and efficient learning compared to reinforcement learning alternatives.

Experimental Evaluation

We evaluated BOBCAT across five diverse, large-scale student response datasets: EdNet (~76M interactions), Junyi (from Junyi Academy), Eedi-1 and Eedi-2 (NeurIPS 2020 Education Challenge), and ASSISTments.

The results demonstrate that BOBCAT is model-agnostic, performing effectively whether paired with traditional IRT models (BiIRT) or complex Deep Neural Networks (BiNN).

Key Results and Insights

BOBCAT consistently outperformed traditional uncertainty-sampling baselines across all datasets in terms of both Accuracy and Area Under the Curve (AUC).

Accuracy comparison across five real-world datasets as a function of the number of questions.

Major findings include:

  • Efficiency Gains: BOBCAT achieved the same predictive accuracy as traditional IRT-Active methods while using up to 50% fewer questions on certain datasets.
  • Superior Ability Estimation: Although not explicitly trained for parameter estimation, BOBCAT produced more accurate student ability estimates ($\theta$) with fewer questions than standard IRT methods.
  • Operational Efficiency: While the bilevel training phase is computationally intensive, the resulting selection algorithm is a simple neural network. At test time, BOBCAT is faster than traditional methods because it avoids solving complex optimization problems between questions.

Limitations and Future Directions

Despite its success, BOBCAT presents challenges for high-stakes environments. The algorithm tends to identify a small subset of highly predictive questions, leading to "question over-exposure" which could compromise test security. Future iterations will need to incorporate exposure control constraints. Additionally, while BOBCAT optimizes for information gain, real-world tests often require content balancing (e.g., specific ratios of math topics), a feature currently absent from the framework. Finally, the reliance on historical data means the model may inherit existing biases or require retraining if the question bank changes significantly.

Summary updated

Abstract

Computerized adaptive testing (CAT) refers to a form of tests that are personalized to every student/test taker. CAT methods adaptively select the next most informative question/item for each student given their responses to previous questions, effectively reducing test length. Existing CAT methods use item response theory (IRT) models to relate student ability to their responses to questions and static question selection algorithms designed to reduce the ability estimation error as quickly as possible; therefore, these algorithms cannot improve by learning from large-scale student response data. In this paper, we propose BOBCAT, a Bilevel Optimization-Based framework for CAT to directly learn a data-driven question selection algorithm from training data. BOBCAT is agnostic to the underlying student response model and is computationally efficient during the adaptive testing process. Through extensive experiments on five real-world student response datasets, we show that BOBCAT outperforms existing CAT methods (sometimes significantly) at reducing test length.
BibTeX
@inproceedings{ghosh2021bobcat,
  author = {Aritra Ghosh and Andrew S. Lan},
  title = {{BOBCAT: Bilevel Optimization-Based Computerized Adaptive Testing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=1zlyyGMMIe},
}
Figure