When we evaluate a system or a human, our default approach is often to use a static benchmark: a fixed set of questions, tasks, or stimuli administered uniformly to every subject. However, static benchmarks are fundamentally inefficient. If an item is too easy or too hard for a given subject, administering it yields almost no new information about their true capability.
To overcome this limitation, researchers across psychometrics, education, and artificial intelligence are leveraging Computerized Adaptive Testing (CAT) and dynamic item selection. Instead of a one-size-fits-all test, an adaptive framework treats evaluation as an iterative, data-driven search. By dynamically selecting the next item based on previous answers, these systems can cut evaluation lengths by 80% to 95% without sacrificing precision.
The Core Intuition: Maximizing Information Gain
To understand why dynamic item selection works, think of it as a probabilistic binary search over a subject's latent trait space (often denoted as $\theta$, representing ability, severity of a symptom, or a model's capability).
In classical psychometrics and Item Response Theory (IRT), every test item has an "information function" determined by its difficulty and discrimination power. The information an item provides is not static; it peaks when the item's difficulty closely matches the subject's current estimated ability.
An adaptive testing loop follows a simple, powerful cycle:
┌──────────────────────────────┐
│ Estimate Latent Ability (θ) │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Select Item Maximize Expected│
│ Information │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Administer Item & Record │
│ Response │
└──────────────┬───────────────┘
│
▼
┌──────────────────────────────┐
│ Stopping Criterion Met? │
└──────┬────────────────┬──────┘
│ │
No │ Yes │
▼ ▼
[Repeat Loop] [Final Score]
By constantly navigating to the boundary of what a subject can and cannot do, the system avoids wasting resources on redundant or uninformative questions.
Porting Psychometrics to AI Evaluation
As Large Language Models (LLMs) and foundation models have grown, evaluating them has become incredibly expensive and prone to data contamination. A rapidly growing body of research argues that AI evaluation should adopt human psychometric techniques to solve these scaling bottlenecks.
Zhuang et al. position this shift as a necessity, demonstrating that tailoring tests to individual AI models drastically reduces the number of items needed for a reliable assessment. In follow-up work, Zhuang et al. show that an adaptive testing perspective allows researchers to reconstruct full benchmark results using only 20% of the original test items.
This efficiency stems from the underlying structure of model capabilities. For instance, Kipnis et al. introduce metabench, revealing that a single "general ability" factor drives the vast majority of LLM performance. Because performance is highly structured, a sparse, dynamic question-selection framework can accurately pinpoint a model's capabilities with minimal steps.
Several specialized implementations validate this approach:
- Fluid Benchmarking: Hofmann et al. introduce an adaptive evaluation strategy that dynamically pairs weaker models with easier items and stronger models with challenging items, maximizing precision while conserving computational resources.
- Clinical Evaluation: Evaluating medical LLMs on massive datasets is notoriously slow. Ouyang et al. apply computerized adaptive testing to clinical scenarios via CliMedBench, maintaining high correlations with full-scale benchmarks while significantly lowering evaluation costs.
Advanced Frameworks for Efficiency and Guarantees
Beyond basic item selection, recent papers focus on optimizing the structural and mathematical properties of adaptive tests to provide formal guarantees and handle complex, multidimensional traits.
Formal Certificates and Low-Rank Frameworks
Traditional benchmarks provide point estimates but lack guarantees on how a model will perform on unseen tasks. To address this, Karuturi et al. propose a Latent Capability Model that treats evaluation as a low-rank matrix completion problem. This allows the system to identify a Minimum Sufficient Evaluation Set (MSES) and issue formal "performance certificates" (prediction intervals), proving that 80% to 95% of standard benchmark items are mathematically redundant.
Optimal Stopping Rules
When should an adaptive test stop? Stopping too early harms reliability, while stopping too late wastes compute. Arviv et al. design an adaptive framework rooted in sequential testing. By dynamically adjusting the amount of data collected based on accumulating statistical evidence, their system stops the evaluation precisely when enough data exists to make a sound decision.
Multidimensional and Hierarchical Search
Real-world abilities are rarely unidimensional. To capture nuanced profiles without exploding computational complexity, Mao et al. utilize full-information item bifactor analysis. This statistical framework simultaneously measures a general trait and specific sub-domains, reducing required test lengths by over 90%. Taking optimization a step further, Yu et al. treat test design as a hierarchical structure search, proving that data-driven optimization of the test structure itself can outperform traditional information-based heuristics.
Applications Across Domains
While AI evaluation is a major driver of recent innovations, the exact same principles of dynamic item selection are creating massive efficiency gains across medicine, psychology, education, and physical engineering.
Healthcare and Mental Health
In patient care, minimizing questionnaire length reduces cognitive fatigue and improves data quality. Payne details how the NIH PROMIS initiative revolutionized subjective health measurement by using CAT to tailor symptom questions in real-time.
In mental health, where long assessments can cause users to disengage, Varadarajan et al. introduce ALBA, an adaptive language-based assessment that dynamically selects questions to screen for depression and anxiety with high accuracy and fewer items. Similarly, McClure et al. develop a multidimensional adaptive test for tracking suicide risk in real-time longitudinal studies, ensuring surveys remain brief enough to maintain high participant compliance.
Education and Vision
In language learning, Zhou et al. combine automatic item generation with adaptive testing to evaluate vocabulary proficiency, reducing the average test length by 81% (from 140 questions down to just 26).
The framework even extends beyond text: Sang et al. develop Visual BFI, an adaptive personality test that replaces text questionnaires with image choices. By using adaptive selection logic based on computer vision features, the system rapidly converges on a user's "Big Five" personality traits while bypassing text-based biases.
Autonomous Systems
Finally, the concept applies to physical engineering validation. Feng et al. apply adaptive testing to Connected and Automated Vehicles (CAVs). Instead of running vehicles through thousands of generic test scenarios, their framework dynamically generates customized testing libraries that adapt to the specific vehicle under test, accelerating safety evaluation by 10 to 100 times.
Key Foundational Work
Advancing User Experience Research to Facilitate and Enable Patient Centered Research: Current State and Future Directions (2013) — contributed 2 concept elements to this direction.
Visual BFI: an Exploratory Study for Image-based Personality Test (2016) — contributed 1 concept elements to this direction.