Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds

John P. Lalor, Hao Wu, Hong YuEMNLP 2019 · DOI

TL;DR

This paper proposes a novel method for fitting Item Response Theory (IRT) models using response patterns generated by ensembles of deep neural networks (DNNs) instead of human annotators. This approach significantly reduces the cost and scalability issues associated with traditional IRT, enabling its application to massive datasets and demonstrating its utility in tasks like training set filtering for more efficient model training.

What Problem Does This Paper Address?

Item Response Theory (IRT) is a powerful psychometric framework for understanding the relationship between latent traits (like ability or difficulty) and observed responses. It's traditionally used to analyze human performance on tests, providing insights into item difficulty and individual ability. However, applying IRT to the massive datasets used for training modern deep neural networks (DNNs) is impractical because it requires human response patterns (RPs) for each item, which is prohibitively expensive and time-consuming for millions of data points. This bottleneck prevents us from leveraging IRT's analytical power for large-scale machine learning datasets.

Methods

The core innovation here is replacing human RPs with RPs generated by an "artificial crowd" of DNN models. Specifically, the authors trained ensembles of LSTM models (for Natural Language Inference and Sentiment Analysis tasks) on subsets of data with corrupted labels. This process generates a diverse set of "responses" from the DNNs, mimicking human RPs.

To fit IRT models to these large-scale, machine-generated RPs, the authors employ Variational Inference (VI). VI is a technique for approximating complex probability distributions, making it suitable for scaling IRT to the size of typical machine learning datasets, unlike traditional methods like Marginal Maximum Likelihood (MML) which struggle with large numbers of subjects. They also compare VI with MML on smaller, human-annotated datasets to validate its accuracy.

Datasets

The paper uses two standard NLP datasets:

  • Stanford Natural Language Inference (SNLI): A dataset for natural language inference tasks, consisting of premise-hypothesis pairs and labels (entailment, contradiction, neutral).
  • Stanford Sentiment Treebank (SSTB): A dataset for sentiment analysis, specifically the SST-2 split for binary sentiment classification (positive/negative).

Human response pattern data for these datasets was previously collected from Amazon Mechanical Turk (AMT) workers.

Key Insights and Ideas

  • DNNs as "Artificial Crowds": The paper demonstrates that ensembles of DNNs can effectively simulate human response patterns for IRT modeling, overcoming the scalability bottleneck of human annotation.
  • IRT for DNNs: This work bridges the gap between psychometrics and deep learning, allowing us to analyze DNN behavior and data using IRT principles.
  • Latent Difficulty for Data Filtering: Learned item difficulty parameters from IRT can be used to intelligently filter training data, leading to more efficient learning and better generalization.
  • Human-Machine Disagreement: The study highlights cases where human and DNN perceptions of item difficulty diverge, offering insights into the different reasoning processes of humans and machines.

Key Results

  • Parameter Correlation: Learned item difficulty parameters from machine RPs showed moderate positive correlations with those learned from human RPs (Spearman $\rho$ around 0.4-0.5 for NLI and SA tasks). More complex DNN architectures (like NSE) yielded higher correlations.
  • VI Validation: Variational Inference (VI) produced IRT parameter estimates very similar to traditional MML methods, with small root mean squared differences (RMSDs around 0.15).
  • Training Set Filtering: Using the AVI (Absolute Value Inner) strategy, which selects "average" difficulty items, achieved high test set accuracy with as little as 25% of the training data for both SNLI and SSTB. This outperformed random sampling and other filtering strategies.
  • Qualitative Analysis: The paper identified specific examples where human and DNN difficulty rankings differed significantly, often due to human intuition or abstract reasoning versus DNNs relying on explicit linguistic cues.

Limitations and Future Work

  • Up-front Cost of DNN Ensembles: Generating RPs from a large ensemble of DNNs still incurs an initial computational cost, though it's argued to be significantly less than human annotation for large datasets.
  • Human-Machine Disagreement: While correlations exist, they are not perfect. The paper acknowledges that understanding why these disagreements occur is an important future direction, potentially involving deeper analysis of model architectures and human cognitive processes.
  • Class Imbalance: The filtering strategies, while effective, could lead to class imbalances in the training set, which the paper notes did not significantly harm performance but warrants further investigation.
  • More Complex Models: The authors suggest that using ensembles of more sophisticated models like BERT could further increase the correlation between human and machine difficulty estimates.
  • Beyond NLP: The methods are applicable to other machine learning domains, such as computer vision, and future work could explore these.
  • Advanced Sampling: Exploring more advanced sampling strategies that combine AVI with outlier incorporation or Bayesian approaches is suggested.

Summary updated

BibTeX
@inproceedings{lalor2019learning,
  author = {John P. Lalor and Hao Wu and Hong Yu},
  title = {{Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1434},
}
Figure