TL;DR
This study investigates the use of ChatGPT for generating multiple-choice questions (MCQs) for an operating systems course. While students couldn't reliably distinguish AI-generated from human-written questions, they performed significantly worse on AI-generated MCQs, suggesting that even validated AI questions may not fully align with instructor intent or student learning styles.
What Problem Does This Paper Address?
Creating effective assessment questions, particularly multiple-choice questions (MCQs), is a time-consuming but crucial task for educators. Instructors need to craft questions that accurately assess student understanding, are fair, and are relevant to the course material. This process is particularly challenging because it requires not only subject matter expertise but also pedagogical skill in designing plausible distractors and unambiguous correct answers. The advent of Large Language Models (LLMs) like ChatGPT presents a potential solution for automating question generation, but it's unclear how these AI-generated questions perform in a real classroom setting, both in terms of student outcomes and their ability to perceive the authorship. This paper aims to bridge this gap by evaluating the impact of LLM-assisted MCQs on student performance and perception.
Methods
The researchers designed an experiment within a third-year operating systems course. For each human-authored MCQ, they attempted to generate a corresponding AI-authored MCQ using ChatGPT. These AI-generated questions underwent a rigorous two-pass validation process by the instructor to ensure they were clear, unambiguous, tested the intended knowledge, and had plausible distractors. Only validated AI-generated questions were used.
The study then randomly deployed either the human-authored or the AI-authored MCQ to students for quizzes and exams. Student performance was measured by their scores on these questions. Additionally, students were asked to rate their perception of whether each question was written by a human or an AI on a 6-point scale. To assess the alignment of questions with course material, cosine similarity was calculated between the question text and the course textbook using SBERT embeddings.
Datasets
The study was conducted in a real-world operating systems course at Southern Illinois University Edwardsville (SIUE). The primary "dataset" consisted of 32 human-authored MCQs, of which 24 had corresponding AI-generated counterparts created using ChatGPT. The course textbook used for similarity comparisons was Andrew Tanenbaum's "Modern Operating Systems, 4th Edition." No new benchmarks were introduced; rather, existing course materials and student performance data were utilized.
Key Insights and Ideas
- AI-generated questions are not easily detected by students: Despite efforts to create distinct question sets, students were unable to reliably distinguish between human-authored and AI-generated MCQs. This suggests that LLMs, when guided by human validation, can produce questions that are superficially similar to those created by instructors.
- Performance gap favors human-authored questions: This is the most striking finding. Students scored significantly lower on AI-generated MCQs compared to human-authored ones, even when both were validated. This indicates that subtle differences in question design, tone, or pedagogical approach, even if not consciously perceived by students, can impact learning outcomes.
- AI questions are more aligned with textbooks: Interestingly, AI-generated questions showed a higher cosine similarity to the course textbook than human-authored questions. This could be because LLMs are trained on vast amounts of text, including textbooks, and may directly pull phrasing or concepts. However, this textbook alignment didn't translate to better student performance, suggesting that effective teaching and assessment go beyond simply mirroring textbook content.
- Instructor's touch matters: The study speculates that the instructor's unique style, emphasis, and pedagogical approach, which are not explicitly captured by AI, might be crucial for student comprehension and performance.
Key Results
- Perception: A Mann-Whitney U test showed no significant difference in students' ability to detect human from AI questions (p = .31).
- Score: Students scored significantly higher on human-authored questions (M = 0.83) compared to AI-generated questions (M = 0.74), with a Mann-Whitney U test indicating a significant difference (p < .01). This represents an almost 9% difference in scores.
- Cosine Similarity: AI-generated questions were significantly more similar to the course textbook (M = 0.21) than human-authored questions (M = 0.14), as shown by a Mann-Whitney U test (p < .001).
Limitations and Future Work
The authors acknowledge several limitations:
- LLM Specificity: The results might be dependent on the specific LLM (ChatGPT-40) and its parameters used.
- Textbook Focus: Cosine similarity was only calculated against the primary textbook, not other course materials like lecture slides or code samples.
- Zero-Shot Prompting: The study used zero-shot prompting, meaning they didn't iterate extensively with the LLM to refine prompts. Further prompt engineering could potentially yield AI questions more aligned with instructor style.
- Bloom's Taxonomy: MCQs, by their nature, may not effectively assess higher-order thinking skills like "create" or "synthesis."
Future work suggested includes:
- Replication with Different Instructors/Disciplines: To control for instructor bias and explore discipline-specific differences in LLM performance.
- Prompt Refinement: Investigating how iterative prompting and fine-tuning LLMs on specific course materials could improve question quality and alignment.
- Broader Course Material Analysis: Including lecture notes, slides, and other resources in similarity calculations.
Summary updated