Predicting Difficulty and Discrimination of Natural Language Questions

Matthew Byrd, Shashank SrivastavaACL 2022 · OpenReview

Abstract

Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education. More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets. In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.

Introduction

In the fields of education and psychometrics, the quality of an assessment item is determined by more than just its binary "hard" or "easy" status. A crucial metric is discrimination: the degree to which a question can distinguish between high-ability and low-ability subjects. In the context of Natural Language Processing (NLP), we typically evaluate models based on their performance across a dataset. However, we rarely evaluate the intrinsic psychometric properties of the questions themselves.

Traditionally, determining a question's difficulty and discrimination requires a live testing environment where hundreds of subjects provide responses. This research presents a methodology to predict these latent traits directly from the text of the question, answer, and context, enabling the "pre-screening" of datasets before they are deployed.

Methodology

The researchers utilized Item Response Theory (IRT), a mathematical framework that models the probability of a correct response as a function of the subject's ability and the item's characteristics.

Creating an "Artificial Crowd"

To circumvent the high cost of human data collection, the authors generated an "artificial crowd" by training 148 instances of the DFGN question-answering model. By varying the training data volume and duration, they created a population of models with a broad spectrum of "abilities."

Feature Engineering and Prediction

The study extracted two categories of features to map text to IRT parameters:

  • Human-Centric Features: These include lexical and syntactic counts, semantic ambiguity (derived from WordNet), and standard readability scores like Flesch-Kincaid.
  • Machine-Centric Features: These include BERT contextual embeddings, n-gram overlap between questions and contexts, and Part-of-Speech (POS) tags.

Regression models, including Random Forests and Gradient Boosted Regressors, were then trained to predict the IRT parameters from these textual features.

Experimental Setup

The research focused on HotpotQA, a dataset designed for multi-hop reasoning. Its inherent complexity provides a more nuanced landscape for studying difficulty and discrimination than simpler datasets. A subset of 4,000 questions was used for the primary analysis.

Results and Analysis

The experiments confirmed that psychometric traits are indeed predictable from raw text. The predictive models achieved a Test $R^2$ of 0.17 for difficulty and 0.13 for discrimination, representing a significant improvement over the baseline mean prediction.

Table 1: Performance of various feature sets in predicting question difficulty. The combination of all features yields the highest predictive power.

Key Insights

  • The Impact of Numeric Data: One of the most significant findings was that the presence of cardinal numbers (CD) in an answer is a strong predictor of high discrimination. This suggests that the underlying QA model (DFGN) struggles with math, and only the most "capable" versions can answer numeric questions correctly.
  • Surface-Level Bias and Clustering: Using UMAP visualizations, the researchers found that questions naturally cluster by style (e.g., Yes/No questions vs. "Which has more..." questions). These clusters exhibit distinct difficulty profiles, suggesting that surface-level phrasing significantly impacts machine performance.
Figure 1: UMAP visualization showing how questions cluster by topic and style, such as Yes/No questions or Entertainment-related topics.
  • BERT vs. Traditional Metrics: While traditional readability scores offered some signal, BERT embeddings were far more effective at capturing the specific nuances that make a question difficult for a machine.

Conclusion and Future Work

This research demonstrates that it is possible to estimate the "IQ" of a question—its difficulty and discriminatory power—using only its textual content. This has profound implications for dataset construction, allowing researchers to ensure a balanced mix of question types. Furthermore, these findings could enhance curriculum learning, where models are trained on questions of increasing difficulty to improve efficiency.

Future work will explore whether these predictive features generalize across different model architectures (e.g., Transformers like GPT-4) and more diverse NLP tasks beyond question answering.

Summary updated

Abstract

Item Response Theory (IRT) has been extensively used to numerically characterize question difficulty and discrimination for human subjects in domains including cognitive psychology and education (Primi et al., 2014; Downing, 2003). More recently, IRT has been used to similarly characterize item difficulty and discrimination for natural language models across various datasets (Lalor et al., 2019; Vania et al., 2021; Rodriguez et al., 2021). In this work, we explore predictive models for directly estimating and explaining these traits for natural language questions in a question-answering context. We use HotpotQA for illustration. Our experiments show that it is possible to predict both difficulty and discrimination parameters for new questions, and these traits are correlated with features of questions, answers, and associated contexts. Our findings can have significant implications for the creation of new datasets and tests on the one hand and strategies such as active learning and curriculum learning on the other.
BibTeX
@inproceedings{byrd2022predicting,
  author = {Matthew Byrd and Shashank Srivastava},
  title = {{Predicting Difficulty and Discrimination of Natural Language Questions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=YRRHrq7mW3},
}
Figure