In human education, we have a clear sense of what makes a task "hard." We expect a student who can solve a differential equation to have mastered basic addition, and we assume a person who can summarize a legal brief can also identify a simple clickbait headline. However, as Large Language Models (LLMs) scale, researchers are discovering that these human-centric hierarchies of difficulty are fundamentally broken when applied to artificial intelligence.
This research direction explores the "difficulty gap": the phenomenon where LLMs exhibit difficulty-invariant performance, often succeeding at complex graduate-level reasoning while failing at foundational tasks that a first-grader could solve.
The Failure of Human Intuition
The most striking finding across recent benchmarks is that human experts are remarkably poor at predicting what an LLM will find challenging. In domains ranging from medical decision-making to video question-answering, there is often near-zero correlation between human-rated difficulty and model performance.
Z. H. Lei and colleagues demonstrated this in GAOKAO-Eval, showing that models frequently fail simple tasks while passing complex ones. This suggests that LLMs do not follow a linear "pedagogical progression" like humans do. Instead, their success is often a function of training data frequency and statistical shortcuts rather than a structured understanding of concepts. For instance, Hasan Abed Al Kader Hammoud found in AraLingBench that models might solve "hard" linguistic tasks simply because those patterns appeared frequently in their training data, while failing "easy" tasks that required genuine structural reasoning.
Measuring Intelligence with Psychometrics
To move beyond unreliable human labels, researchers are borrowing tools from the field of psychometrics, specifically Item Response Theory (IRT). IRT allows researchers to estimate the "latent ability" of a model and the "true difficulty" of a question based on empirical pass rates rather than subjective judgment.
Aakriti Agrawal and the team behind Easy2Hard-Bench used these models to create a "high-resolution map" of LLM performance. They found that while mathematical difficulty is somewhat consistent between humans and models, linguistic and commonsense reasoning tasks show a massive divergence. Similarly, C.C. Pan introduced AutoJudger, which uses IRT to select the most "informative" questions for a model, significantly reducing evaluation costs by ignoring questions that don't align with the model's specific knowledge frontier.
The Curse of Knowledge and Expert Blindness
As models become more powerful, they don't necessarily become better at understanding human struggles. In fact, they may become worse. Ming Li identified a "Curse of Knowledge" in Can LLMs Estimate Student Struggles?, where high-performing models converge toward a "machine consensus" of difficulty. These models solve complex tasks so effortlessly that they lose the ability to simulate the cognitive hurdles or misconceptions that a human beginner faces.
This "expert blindness" has practical consequences for AI-driven education and triage. If a model cannot accurately estimate task difficulty, it cannot effectively act as a tutor or a triage agent. For example, Cynthia Breazeal and the MDAgents team found that AI-driven medical triage systems prioritize resources differently than human practitioners because their perception of "hard cases" is fundamentally misaligned with clinical reality.
Why the Gap Exists
Why do these models fail at "easy" things? The research points to several structural bottlenecks:
- Training Bias: Models are "super-experts" in areas overrepresented in their training data but "novices" in niche or low-resource areas, regardless of the task's inherent logical complexity (AraLingBench).
- Brittle Heuristics: Models often use "Pseudo Aha Moments"—logically unsound shortcuts—to solve problems. When a problem is modified to break these shortcuts, the model's "reasoning" collapses (EvolMathEval).
- Cognitive Bottlenecks: Humans are limited by working memory and "cognitive intensity." LLMs appear largely indifferent to these constraints, allowing them to bypass the bottlenecks that make certain science problems hard for students (Xiaoming Zhai in Can generative AI and ChatGPT outperform humans?).
Toward Machine-Centric Benchmarking
The consensus across these 83 papers is that we must stop evaluating AI using human yardsticks. Bruce W. Lee argues in Instruction Tuning with Human Curriculum that even our training data order should be reconsidered; what we call "simple-to-complex" in a classroom may not be the optimal "pedagogical progression" for a neural network.
By acknowledging that machine complexity is distinct from human complexity, researchers can build more robust benchmarks like LogiEval or MATH-Beyond that target the specific structural failures of LLMs, rather than just reciting the contents of a high school textbook.