How do we determine if a question, an exam item, or a text snippet is "hard"?
Traditionally, answering this requires empirical data. In educational testing and machine learning evaluation alike, an item's difficulty is calibrated post-hoc by observing how a population of students or models performs against it. If 90% of test-takers fail a question, it is hard; if 90% pass, it is easy.
While highly accurate, this empirical approach suffers from a fundamental bottleneck: the cold-start problem. When a test developer writes a new question, or when an AI researcher introduces a new evaluation benchmark, there is zero historical interaction data available. Collecting this data via pretesting is slow, expensive, and logistically complex.
A rapidly growing body of research solves this problem through amortized difficulty estimation. The core intuition is simple: the text of a task contains intrinsic, latent signals—such as syntactic complexity, conceptual depth, and semantic density—that dictate its difficulty. By training language models to extract these features, we can predict an item's empirical difficulty at the exact moment of its creation, completely bypassing the need for human or model pretesting.
The Mechanism: Mapping Text to Latent Complexity
To understand why text-based difficulty estimation works, we have to look at how text is transformed into a predictive signal. The process typically follows a two-stage pipeline: semantic encoding and parameter mapping.
[ Question Text ] ---> [ Text Encoder / LLM ] ---> [ Dense Embedding ] ---> [ Mapping Function ] ---> [ Predicted Difficulty ]
First, a text encoder (ranging from smaller models like BERT to massive Large Language Models) processes the raw text of a question or task. This step converts linguistic nuances into a dense vector embedding. As Daumé III et al. demonstrate, these content-aware embeddings are capable of capturing not just keywords, but the underlying skill requirements and cognitive dimensions of a task.
Second, this embedding is passed to a mapping function trained to predict a target difficulty metric. Depending on the framework, this metric can take several forms:
- Empirical Pass Rates: Predicting the exact proportion of users who will answer an item correctly, as explored by Shi et al..
- Continuous Spectrum / Ordinal Ranks: Treating difficulty as a continuous scale rather than isolated categories, often optimized via ordinal regression models like ORMS-BERT proposed by Wu et al..
- Latent IRT Parameters: Mapping text directly to the mathematical parameters of Item Response Theory (IRT).
The surprising efficacy of this approach lies in its efficiency. Truong et al. show that a simple linear model stacked on top of frozen text embeddings can accurately predict latent IRT parameters. This "amortized" approach matches the performance of traditional calibration methods that require multiple models to answer every single question, proving that semantic features alone contain sufficient signal to estimate task complexity.
From Surface Heuristics to LLM Judgments
Early attempts at automated difficulty estimation relied heavily on surface-level textual complexity features. For instance, Loukina et al. demonstrated that automated systems using linguistic complexity metrics could predict the difficulty of listening comprehension items at levels comparable to human experts.
However, surface heuristics like sentence length or word frequency break down when a task's difficulty stems from abstract logic rather than complex vocabulary. To capture deeper conceptual relationships, recent frameworks leverage the zero-shot and fine-tuned capabilities of Large Language Models (LLMs).
Because LLMs are trained on vast corpora of human knowledge, they possess an internal understanding of which concepts are elementary and which are advanced. Wong et al. show that powerful teacher models can serve as highly effective difficulty estimators, achieving strong alignment with human expert assessments of linguistic and grammatical complexity. Similarly, Lee et al. utilize fine-tuned LLMs to dynamically output difficulty scores for educational concepts, moving the field away from static, manual assignments toward flexible, automated content understanding.
Real-World Applications
Predicting difficulty directly from text unlocks significant capabilities across several domains by turning a slow, empirical process into an instantaneous inference step.
1. Cold-Start Knowledge Tracing
In adaptive learning platforms, knowledge tracing models track a student's mastery over time to recommend the next best exercise. When a new question is added to the database, the system traditionally cannot use it effectively until dozens of students have attempted it. By predicting exercise difficulty directly from text features, frameworks can integrate new content into the system immediately, solving the cold-start problem for personalized education (Tong et al.; Lee et al.).
2. Automated Curriculum Learning
When training machine learning models on complex generation tasks like abstractive text summarization or grammatical error correction, the order of training data matters. Training is more stable and effective when models progress from "easy" to "hard" examples. Rather than relying on naive proxies like text length, researchers use LLMs to evaluate the semantic difficulty of training instances, structuring highly nuanced curricula that improve downstream model performance (Li et al.; Wong et al.).
3. Test-Time Compute Optimization
For LLMs performing complex reasoning, techniques like self-consistency generate multiple reasoning paths and vote on the final answer. However, generating dozens of paths for an easy question wastes massive amounts of compute. As Liu et al. (2026) note, if a system can predict a question's reasoning difficulty straight from its text before running any inference, it can dynamically allocate its compute budget—generating many paths for hard questions and saving resources on easy ones.
Key Foundational Work
Question Difficulty Estimation in Community Question Answering Services (2013) — contributed 1 concept elements to this direction.
Textual complexity as a predictor of difficulty of listening items in language proficiency tests (2016) — contributed 1 concept elements to this direction.