Designing high-stakes standardized tests requires knowing exactly how difficult each question is before it is administered. Traditionally, test developers rely on two methods: expensive pre-testing (field trials with large student cohorts) or subjective evaluation by subject matter experts. As noted by Beinborn et al., manual estimation is highly subjective and error-prone. Experts frequently suffer from an "expert blind spot"—they struggle to accurately calibrate their professional intuition against the actual cognitive capabilities and historical performance of diverse student populations (Shang et al.; Kapoor et al.).
This mismatch introduces inconsistency across test forms, potentially compromising fairness. Automated estimation aims to bridge this gap by extracting objective signals of difficulty directly from the test items.
From Surface Text to Deep Semantics
The earliest automated approaches treated question difficulty as a function of linguistic complexity. By analyzing surface-level features—such as syntactic structure, sentence length, and vocabulary level—models could approximate the cognitive load required to process an item. For instance, Loukina et al. demonstrated that text complexity features could predict the difficulty of listening comprehension items with an accuracy matching or exceeding human estimates. Similarly, Skidmore et al. leveraged multilingual transformer architectures to capture vocabulary difficulty for second-language learners.
However, text complexity alone fails when a question's difficulty stems from conceptual reasoning rather than reading comprehension. To capture these deeper interactions, researchers shifted toward architectures that explicitly model the relationship between the question, the options, and the source text. Huang et al. introduced TACNN (Test-aware Attention-based Convolutional Neural Network), which uses attention mechanisms to determine how different parts of a reading passage inform a specific question, outperforming high school teachers in predicting item difficulty.
Large Language Models as Proxy Students
The advent of Large Language Models (LLMs) enabled a paradigm shift: instead of treating difficulty prediction as a static regression task based on text features, researchers began using LLMs to simulate the test-taking population. The core intuition is that an item's difficulty is not an intrinsic property of the text; it is an emergent property of the interaction between the item and the test-taker.
Under the LLaSA (LLMs are Students At various levels) framework proposed by Park et al., a diverse pool of LLMs acts as a proxy student body. By evaluating how models of varying capacities perform on a given question, the system can infer human difficulty in a zero-shot setting without requiring empirical student data. Similarly, Duenas et al. simulated test-takers under varying conditions to predict the difficulty of complex medical multiple-choice questions, demonstrating that simulating the population yields a more accurate reflection of difficulty than direct prompt-based estimation.
Quantifying Difficulty through Model Uncertainty
If LLMs behave like proxy students, their internal state during the answering process can reveal the hidden hurdles within a question. Recent work leverages model uncertainty as a direct proxy for human struggle.
When an LLM evaluates a multiple-choice question, its confidence can be quantified via first-token probabilities or its sensitivity to distracting options. Zotos et al. showed that combining these internal uncertainty metrics with traditional text features yields state-of-the-art difficulty predictions across medical and reading comprehension datasets.
In a follow-up study, Zotos isolated specific behavioral markers of uncertainty, such as how much a model's answer changes when the order of multiple-choice options is shuffled. When a model exhibits high entropy or high sensitivity to option ordering, it indicates that the distractors are cognitively competitive—a phenomenon that closely mirrors why human students find certain items difficult.
Feature Engineering and Hybrid Pipelines
While deep learning and LLMs offer powerful behavioral proxies, the most robust systems often combine these insights with structured linguistic and physical features. In the context of specialized domains like medical licensing exams, Ram et al. demonstrated that combining textual semantic embeddings with physical features (such as question length and numerical density) improves both difficulty and response time estimation.
Furthermore, Veeramani et al. built a hybrid pipeline integrating Named Entity Recognition (NER) and Semantic Role Labeling (SRL) with LLM features to capture domain-specific complexity. Surprisingly, as shown by Felice et al., carefully engineered features fed into simple ensembled linear regression models can compete with or outperform pure transformer-based architectures, highlighting that item difficulty remains deeply rooted in identifiable linguistic and structural properties.
Key Foundational Work
Textual complexity as a predictor of difficulty of listening items in language proficiency tests (2016) — contributed 1 concept elements to this direction.
Candidate evaluation strategies for improved difficulty prediction of language tests (2015) — contributed 1 concept elements to this direction.