TL;DR
This paper explores using large language models (LLMs) like GPT-3 to automatically generate test items that are not only comparable to human-written items but potentially better in terms of psychometric properties. They introduce a novel prompting strategy that leverages item discrimination to guide LLM generation, showing promising results for improving validity and reliability, especially for constructs with limited prior psychometric analysis.
What Problem Does This Paper Address?
Creating high-quality test items for cognitive assessments is a crucial but notoriously time-consuming and labor-intensive process. While AI, particularly LLMs, offers a promising avenue for automated item generation (AIG), much of the existing research has focused on generating items that merely match the psychometric properties of well-established, human-validated items. This paper tackles a more challenging scenario: can LLMs generate items that improve upon existing ones, especially when the original items have weak validity evidence or are for a less-studied construct? This is vital for accelerating the development of new assessments and iteratively refining existing ones.
Methods
The core of the paper's methodology lies in a novel prompting strategy for GPT-3. Instead of simply providing examples of good items, they employ a "simple" prompt structure that includes both high-discrimination and low-discrimination examples of human-written items from the target construct. This "good and bad" example approach aims to teach the LLM the general characteristics of high-quality items, with a specific focus on improving item discrimination. They also experimented with "elaborated" prompts that provided more contextual information about the specific Natural Language Inference (NLI) categories.
For evaluating the generated items, they employed a multi-faceted approach:
- Content Review: Ph.D. students with relevant expertise rated the generated items on relevance, clarity, and potential for harmful content.
- Human Studies: Participants recruited via Amazon Mechanical Turk completed both human-written and GPT-3 generated items.
- Psychometric Analysis: Classical Test Theory (CTT) was used to analyze item difficulty, item discrimination (item-to-total correlation), internal consistency reliability (Cronbach's alpha), and convergent/discriminant validity using a Multi-Trait Multi-Method (MTMM) matrix.
- Local Item Dependency (LID) Analysis: They investigated whether LLM-generated items introduced more LID than human-written items.
Datasets
The primary dataset used for this research is the AX diagnostic set from the General Language Understanding Evaluation (GLUE) benchmark. AX consists of Natural Language Inference (NLI) items, each presented with a premise and a hypothesis, requiring a model to determine if the premise entails, contradicts, or is neutral with respect to the hypothesis. The AX dataset was chosen because its items, while having some evidence of validity, have not undergone extensive psychometric development, making it suitable for testing the LLM's ability to improve item quality.
Key Insights and Ideas
- LLMs can improve item quality, not just replicate it: The most significant insight is that LLMs, when guided by a strategic prompting approach, can generate items that exhibit superior psychometric properties compared to human-written counterparts, even for constructs with limited prior analysis.
- Discrimination is a powerful optimization target: Focusing on item discrimination in the prompt proved effective in guiding GPT-3 to produce items that better differentiate between test-takers.
- "Good and bad" examples are key: Providing examples of both high and low discrimination items in the prompt helps the LLM learn the nuanced characteristics of effective test items.
- Construct scope matters: The benefits of LLM-generated items were more pronounced for narrowly scoped constructs (like PS and Q) compared to broader ones (like MN and LE), suggesting that LLMs might struggle with highly complex or ambiguous constructs without further refinement.
- GPT-3 is a viable tool for iterative item improvement: The findings suggest that LLMs can be integrated into a workflow for continuously refining and enhancing assessment items.
Key Results
- Improved Discrimination and Reliability: GPT-3 generated items consistently showed higher item discrimination and internal consistency reliability compared to human-written items across most categories.
- Comparable Validity: GPT-3 items maintained strong convergent, discriminant, and content validity, often matching or exceeding human-written items.
- Item Difficulty: GPT-3 items were closer to the optimal difficulty level (around 70% accuracy) than human-written items.
- Local Item Dependency (LID): LLM-generated items did not appear to worsen LID compared to human-written items, and in some cases, showed distributions indicating fewer LID issues.
- GPT-4 Performance: Preliminary analysis with GPT-4 showed mixed results, suggesting that prompt engineering might need to be adapted for newer models.
Limitations and Future Work
The authors acknowledge several limitations:
- Small Sample Size: The study used a relatively small sample of human participants, limiting the generalizability of the findings and preventing analyses like internal structure or differential item functioning (DIF).
- Item Diversity: The study did not explicitly examine the diversity of generated items, which could lead to issues like having too many similarly worded items.
- Unbalanced Item Distribution: The generated items showed some imbalance in label distribution (e.g., fewer neutral items), which could be addressed with improved prompting.
- GPT-4 Adaptation: The preliminary GPT-4 results were mixed, indicating a need for further research into adapting prompts for newer LLMs.
- Potential for Bias: LLMs can generate biased or harmful content, and while content review was performed, further work is needed to ensure fairness and mitigate potential disadvantages for certain groups.
Future work should focus on larger sample sizes for more robust psychometric analyses, developing prompting strategies that balance item quality with diversity, exploring DIF, and adapting the approach for more advanced LLMs like GPT-4. The authors also suggest exploring tasks beyond NLI that are more directly applicable to educational settings.
Summary updated