TL;DR
This paper introduces BERT-IRT, a novel approach that leverages BERT embeddings and engineered NLP features to significantly accelerate the process of estimating item parameters for language proficiency tests. By reducing the need for extensive pilot testing, BERT-IRT promises to lower costs, improve test security, and make assessments more accessible.
What Problem Does This Paper Address?
Modern high-stakes tests, especially those using Computer Adaptive Testing (CAT), rely on accurate item parameters (like difficulty and discrimination) to function effectively. Traditionally, these parameters are estimated through lengthy pilot studies where a large number of test-takers respond to new items. This process is costly, time-consuming, and limits how frequently item banks can be updated, potentially impacting test security and fairness. The paper addresses the challenge of reducing the reliance on these extensive pilot studies by developing a method that can estimate item parameters more efficiently using item features.
Methods
The core of the paper is the BERT-IRT model, which builds upon existing Explanatory Item Response Theory (IRT) frameworks. Unlike traditional IRT models that solely rely on response data, BERT-IRT uses item features to predict parameters. The novelty lies in:
- BERT Embeddings: The model utilizes BERT embeddings, specifically from the first four layers, to capture rich linguistic properties of words within test items.
- Engineered NLP Features: In addition to BERT embeddings, the model incorporates 15 hand-crafted NLP features (e.g., word frequency, key length, vowel proportion) that have been shown to correlate with item difficulty.
- Explainable IRT Framework: BERT-IRT models item parameters (intercept for difficulty, slope for discrimination) as linear functions of these features.
- Residual Weights: To balance the predictive power of features with the accuracy derived from actual response data, BERT-IRT uses residual weights. This allows the model to refine feature-based estimates using observed responses, akin to Bayesian updating.
- Testlet Scoring: The authors treat each damaged word in a C-test as a distinct item within a testlet, which significantly improves criterion validity and reliability compared to treating the entire passage as a single item.
Datasets
The study primarily uses historical response data from the Duolingo English Test (DET), a high-stakes English proficiency test. This data consists of C-test passages, with millions of responses from test-takers. The authors also generated new C-test passages using GPT-3 for specific experiments.
Key Insights and Ideas
- BERT as a powerful feature extractor: BERT's ability to capture nuanced linguistic properties is crucial for predicting item difficulty and discrimination, outperforming simpler feature sets.
- Feature engineering still matters: While BERT embeddings are powerful, combining them with engineered NLP features leads to superior performance, suggesting a complementary relationship.
- Residual weights bridge the gap: The inclusion of residual weights is key to achieving high accuracy, especially for items with ample response data, by allowing the model to adapt feature-based predictions.
- Testlet scoring enhances validity: Treating individual damaged words as items within a testlet significantly boosts the reliability and validity of IRT models for C-tests.
- Accelerated piloting is feasible: The core idea that BERT-IRT can drastically reduce pilot study duration without sacrificing critical validity and reliability metrics is a major takeaway.
Key Results
- Significant reduction in piloting time: BERT-IRT enabled item piloting with only a tenth of the normal pilot data, reducing pilot duration by a factor of 10.
- Comparable or better performance with less data: BERT-IRT achieved similar or better performance than traditional operational 2PL models with as few as 50 responses per item, a 4x increase in piloting efficiency.
- Strong generalization: The model demonstrated good generalization of item parameter predictions to unseen items, even when using different data splits (testlet and item splits).
- Feature importance: Word embeddings contributed significantly (28-40%) to prediction accuracy, while word frequency features were also highly influential, sometimes even more so than embeddings for intercept prediction.
Limitations and Future Work
- Estimating variance explained by features: The authors acknowledge that directly estimating the variance in item parameters explained by features could further improve the model, but finding a tractable method for a large number of features is challenging.
- C-test specific: The study focused on C-tests. Future work will involve evaluating BERT-IRT and similar NLP-driven IRT models on other item types.
- CAT efficiency impact: While the study suggests fewer pilot responses are needed, the potential impact of less accurate parameters on CAT algorithm efficiency requires further study.
- Future directions: The authors plan to explore applications to other item types and further reduce or eliminate the need for item piloting.
Summary updated