TL;DR
This paper investigates whether ChatGPT can generate explanations for reading comprehension multiple-choice questions that are comparable in quality to those written by human experts. The study found that ChatGPT's explanations are often more detailed and can be a valuable resource for improving human-written explanations, suggesting AI's potential to assist in educational assessment development.
What Problem Does This Paper Address?
Developing high-quality multiple-choice questions for reading comprehension is a meticulous process. It requires not only crafting plausible options and a clear stem but also providing thorough explanations for why each option is correct or incorrect. These explanations are crucial for understanding the item's rationale, serving as valuable feedback for learners, and even revealing subtle flaws in the item itself. However, generating these explanations is time-consuming, often leading to items lacking them. This gap highlights the need for tools that can assist in this process. While AI language models like ChatGPT have shown promise in various domains, their efficacy in generating explanations for educational assessments, specifically in reading comprehension and in Spanish, has not been extensively explored. This paper aims to bridge this gap by comparing ChatGPT's explanations with those created by human item-writers.
Methods
The researchers employed a comparative approach, pitting ChatGPT against human item-writers. They utilized a dataset of multiple-choice reading comprehension items from Colombia's "Evaluar para Avanzar" (EpA) strategy, which is based on Evidence-Centered Design (ECD). This design framework ensures that items are linked to specific learning claims and evidence.
The study involved:
- Data Preparation: Manually copying contexts and items from EpA booklets and cleaning them.
- Human Explanation Extraction: Extracting explanations for each option from the existing human-authored materials.
- ChatGPT Explanation Generation: Prompting ChatGPT with item contexts and questions to generate explanations for each option. A specific methodology was used to ensure ChatGPT's memory was focused on the relevant context and item.
- Comparative Analysis: A linguist and native Spanish speaker manually compared pairs of human and ChatGPT explanations using a six-color tagging system (MIC, AII, AIC, EIC, CI, CC) to categorize similarities and differences in content, expansion, and closure.
Datasets
The study used a dataset of human-written textual explanations for multiple-choice reading comprehension items from the "Evaluar para Avanzar" (EpA) strategy in Colombia. These items were drawn from grades 5°, 9°, and 11° for the academic years 2021-2 and 2022-1. The dataset comprised a total of 88 items, with contexts varying in text type (expository, narrative, descriptive, argumentative) and length. Items requiring image context were excluded as ChatGPT does not support this format.
Key Insights and Ideas
- ChatGPT as a Detailed Explainer: ChatGPT consistently produced more verbose explanations than humans, often providing additional context or elaborating on the reasoning. This suggests AI can offer a richer explanatory experience.
- Complementary Strengths: While humans tend to be more direct and may omit closure statements, ChatGPT often includes them, sometimes repeating the rationale in different words. This indicates a potential for humans to learn from ChatGPT's thoroughness.
- Potential for AI-Assisted Development: The findings suggest that ChatGPT can be a valuable tool for assisting educators and test developers in creating high-quality assessment items, potentially increasing both the quantity and quality of explanations.
- ECD Classification Promise: ChatGPT demonstrated a promising ability to classify items according to the ECD principles (claims), though further research is needed to refine this capability.
Key Results
- Accuracy: ChatGPT correctly answered 78 out of 88 items. Ten items were excluded due to ChatGPT's responses not aligning with established answers or its inability to provide a conclusive explanation.
- Explanation Length: ChatGPT's explanations were generally longer than human explanations, particularly for additional information (AIC tag), showing a large effect size. However, for common information (MIC tag), the difference in length was significant but with a small effect size.
- Focus of Explanations: Humans tended to provide more additional explanations for correct options, while ChatGPT focused more on incorrect options.
- Classification Accuracy: ChatGPT correctly classified 49 (55.68%) of the items into the three ECD claims. A significant number of misclassifications were observed, particularly for items involving quoted expressions, suggesting a potential misinterpretation of literal text.
Limitations and Future Work
The authors acknowledge several limitations:
- Manual Comparison: The comparative analysis was conducted by a single linguist, introducing potential subjectivity.
- Item Exclusion: Ten items were excluded due to ChatGPT's incorrect answers or inability to provide accurate explanations, limiting the scope of the analysis.
- Misclassification Analysis: A deeper dive into why ChatGPT misclassified certain items (especially those involving quoted text) was beyond the scope of this study and requires further investigation.
- "Which NOT" Questions: ChatGPT sometimes struggled with "Which NOT" questions, either failing to identify the correct option or providing explanations that were not entirely conclusive for incorrect options.
Future work suggested includes:
- Combining Strengths: Exploring ways to integrate the strengths of human expertise and AI to produce even more accurate and informative explanations.
- Prompt Engineering: Developing well-designed prompts to guide AI in creating items with single correct answers, independent options, and plausible distractors.
- ECD Classification Refinement: Providing AI with more explicit guidance on ECD principles and asking it to classify items or generate explanations based on specific evidence.
- Addressing "NOT" Questions: Further research into how AI handles negation and complex question structures in assessment.
Summary updated