When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty

Fernando Martínez-Plumed, David Castellano Falcón, Carlos Monserrat Aranda, José Hernández-Orallo

In the contemporary landscape of artificial intelligence, the primary yardstick for success is aggregate accuracy. We celebrate when a model hits 98% on a benchmark, but we rarely pause to ask: Is the remaining 2% actually difficult, or is our model simply flawed? A recent study presented at AAAI-22, "When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty," addresses this gap by borrowing methodologies from psychometrics to quantify "instance difficulty." By predicting how hard a specific problem is before a model even attempts it, we can develop AI that is more robust, explainable, and trustworthy.

The Problem with Aggregate Metrics

Standard AI evaluation typically relies on mean accuracy or error rates across a dataset. However, these metrics are "black boxes" that obscure the "why" behind model failure. If a vision model misclassifies a digit, was it due to motion blur, an unusual stroke, or a systemic bias in the model's training?

To provide a more granular view, researchers have turned to instance difficulty. While previous attempts often defined difficulty as the average error across a pool of models, this approach is inherently unstable and dependent on the specific systems used. This paper proposes a more rigorous alternative: Item Response Theory (IRT).

Methodology: Borrowing from Psychometrics

IRT is the mathematical framework used to design standardized tests like the SAT or GRE. It models the probability of a correct response as a function of both the test-taker's "ability" and the item's "difficulty."

Item Response Theory (IRT) curves showing the relationship between probability of a correct response, latent ability, and instance difficulty.

The authors implement a two-step "meta-learning" pipeline:

  1. Ground Truth Calculation: They run a large ensemble of AI models across 18 diverse benchmarks. Using the 1PL and 2PL IRT models, they derive a stable, system-independent difficulty parameter ($b_i$) for every individual instance.
  2. Difficulty Estimation: They then train regression models to predict this $b_i$ value using only the raw features of the instance. This creates a "difficulty estimator" that can evaluate new, unseen data instantly without needing to run it through dozens of other AI systems.

Experimental Setup and Datasets

The study spans 18 benchmarks across four distinct domains to test the generalizability of the approach:

  • Tabular Data: Classic supervised learning sets like diabetes and liver-disorders.
  • Computer Vision: Ranging from simple OCR (MNIST, letter) to more complex image recognition (CIFAR-10, Fashion-MNIST).
  • Natural Language Processing: Sentiment analysis tasks using the IMDb and SST-2 datasets.
  • Automated Reasoning: Boolean Satisfiability (sat) and Theorem Proving (tptp).

Key Results: Is Difficulty Predictable?

The core finding is that for most domains, instance difficulty is remarkably predictable from raw features. In many instances, it is actually easier to predict how "hard" a problem is than it is to solve the problem itself.

Performance of various difficulty estimators (nRMSE and correlation r) across tabular, image, and text datasets.

As shown in the tables above, Random Forests (RF) emerged as the dominant estimator for tabular and OCR data, achieving Spearman correlations ($r$) as high as 0.89 to 0.91. For image datasets like MNIST and UMIST, pre-trained CNNs (VGG16) provided highly accurate estimations with normalized Root Mean Squared Errors (nRMSE) as low as 0.34.

However, the "NLP Gap" remains a challenge. Sentiment analysis tasks (IMDb/SST-2) and complex image sets (CIFAR-10) proved more resistant to difficulty estimation. The authors attribute this to the high complexity of the underlying features and, for NLP, a lack of sufficient instance-wise performance data to train robust estimators.

Beyond Accuracy: New Insights

The ability to predict difficulty opens several new avenues for AI research:

  • System Characteristic Curves (SCCs): By plotting accuracy against predicted difficulty, researchers can visualize a model's "breaking point." This reveals that some state-of-the-art models are highly robust on easy tasks but fail catastrophically on difficult ones, whereas older models might degrade more gracefully.
  • Adversarial Detection: A significant mismatch between a low predicted difficulty and a high actual error can act as a "tripwire" for detecting adversarial attacks or "Clever Hans" models that rely on spurious correlations.
  • Feature Divergence: Interestingly, the features that make a task hard to solve are not always the ones that make its difficulty easy to predict. In letter recognition, for instance, "mean edge count" is a primary predictor of difficulty, even if it isn't the main feature used for classification.

Conclusion and Future Directions

By shifting the focus from "how well did we do?" to "how hard was the test?", this research provides a vital new lens for AI evaluation. While challenges remain in complex domains like NLP, the success of IRT-based difficulty prediction suggests a path toward more transparent and accountable AI. Future work will likely focus on extracting better latent features from deep learning models and encouraging the community to share raw, instance-wise results to overcome data scarcity.

Summary updated

Abstract

One of challenges of artificial intelligence as a whole is robustness. Many issues such as adversarial examples, out of distribution performance, Clever Hans phenomena, and the wider areas of AI evaluation and explainable AI, have to do with the following question: Did the system fail because it is a hard instance or because something else? In this paper we address this question with a generic method for estimating IRT-based instance difficulty for a wide range of AI domains covering several areas, from supervised feature-based classification to automated reasoning. We show how to estimate difficulty systematically using off-the-shelf machine learning regression models. We illustrate the usefulness of this estimation for a range of applications.
BibTeX
@inproceedings{martinezplumed2022when,
  author = {Fernando Martínez-Plumed and David Castellano Falcón and Carlos Monserrat Aranda and José Hernández-Orallo},
  title = {{When AI Difficulty Is Easy: The Explanatory Power of Predicting IRT Difficulty}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i7.20739},
}
Figure