TL;DR
This paper introduces a novel pipeline that leverages Large Language Models (LLMs) to automatically predict the difficulty and response time of educational assessment items. By integrating Named Entity Recognition, Semantic Role Labeling, and linguistic features, the framework achieves state-of-the-art results on the BEA 2024 Shared Task and demonstrates adaptability to oral assessments, promising to streamline test development and enhance fairness.
What Problem Does This Paper Address?
Developing standardized tests is a meticulous process, requiring careful estimation of item difficulty and appropriate allocation of response time. Traditionally, this involves labor-intensive pretesting, where new items are embedded in live exams. This method is not only time-consuming but also poses security risks due to potential item overuse. The BEA 2024 Shared Task specifically aims to address this gap by exploring automated prediction methods for these crucial assessment characteristics. The authors highlight the need for more efficient and secure approaches, especially for high-stakes exams, and propose a solution that leverages the advanced language understanding capabilities of LLMs.
Methods
The core of this work is a novel pipeline that combines several NLP techniques to predict item difficulty and response time. The methodology can be broken down into these key steps:
- Named Entity Recognition (NER): The pipeline first identifies named entities within the test item text. It employs a dual-model strategy, using both Longformer (for long document understanding) and a choice of LLMs (Mistral-7B, Llama-7B, or Gemma-7B) to ensure a comprehensive extraction of entities.
- Semantic Role Labeling (SRL): Following NER, SRL is used to identify the semantic relationships between entities and the actions or states described in the sentence. This is achieved using AllenNLP's SRL model and an LLM, providing a deeper understanding of the question's structure and content.
- Difficulty Prediction: The LLM is then prompted to estimate item difficulty. It's fed the input sentence, the identified named entities, and their semantic roles, along with the answer options. The LLM is tasked with summarizing the association between the input context and each answer option, assigning a score based on the perceived difficulty of these linkages.
- Response Time Prediction: For response time, the pipeline incorporates additional linguistic features (numerical and syntactic features from LingFeat) alongside the NER and SRL outputs. The LLM is prompted to consider all these inputs to predict the response time, with a specific range provided for higher values indicating longer times.
The entire process is orchestrated using the Langchain library, facilitating the chaining of API calls to LLMs and post-processing.
Datasets
The primary dataset used is from the BEA 2024 Shared Task, which comprises 667 retired questions from USMLE Steps 1, 2 CK, and 3. This dataset includes item text (stem), answer options, correct answer key, item type, exam step, numerical item difficulty, and average response time.
As an auxiliary task to evaluate the framework's adaptability, the authors also used audio recordings from the Test of Narrative Language (TNL). This dataset consists of oral responses from 185 3rd-8th grade students from Atlanta, Georgia, who were asked to describe an image and tell a story. For this dataset, the authors generated ASR transcripts and used metrics like transcription word accuracy and proportion of correct responses for difficulty, and analyzed disfluencies for response time.
Key Insights and Ideas
- LLMs as powerful predictors: The study strongly suggests that LLMs, when appropriately prompted and augmented with structured linguistic information (NER, SRL, and linguistic features), are highly effective at predicting both item difficulty and response time.
- Synergy of NLP techniques: The combination of NER and SRL with LLMs is crucial. NER provides the "who" and "what," SRL clarifies the "how" and "why," and together they offer a richer context for the LLM to make more informed predictions than using text alone.
- Adaptability across modalities: The framework's successful application to both written (USMLE) and oral (TNL) assessment data demonstrates its flexibility and potential for broader use in educational assessment.
- Handcrafted features still matter: Even with powerful LLMs, the inclusion of carefully selected linguistic features (especially for response time) significantly aids in understanding the complex interplay of language and test-taker behavior.
Key Results
On the BEA 2024 Shared Task, the proposed pipeline achieved an RMSE of 0.308 for item difficulty and 27.474 for response time prediction. These results represent an improvement over the provided baseline. Specifically, prompting Llama2-7B and Gemma-7B outperformed the baseline and other LLMs.
For the Test of Narrative Language (oral assessments), Gemma-7B (Run 3) demonstrated superior performance, achieving an RMSE of 2.007 for item difficulty and 2.022 for response time. This highlights the framework's effectiveness even with transcribed speech data.
Limitations and Future Work
The authors acknowledge several limitations:
- Model Interpretability: The decision-making process of LLMs lacks transparency. Future work should focus on developing methods to increase interpretability and provide human-understandable explanations for predictions.
- Linguistic Feature Scope: The current implementation uses a specific set of linguistic features for response time prediction. Exploring additional domain-specific vocabulary or other features could further enhance accuracy.
- Domain Specificity: While promising across different formats, the framework's performance might vary across diverse domains and test types. Further research is needed to evaluate and adapt it for optimal performance in various testing contexts.
Future work will aim to address these limitations to improve the framework's accuracy, efficiency, and fairness in educational assessments.
Summary updated