Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning

Dan Carpenter, Wookhee Min, Seung Lee, Gamze Ozogul, Xiaoying Zheng, James LesterNAACL 2024

TL;DR

This paper explores using large language models (LLMs) to automatically assess student explanations in undergraduate computer science courses, addressing the scalability issue of manual grading. They found that fine-tuning FLAN-T5 with rubric and exemplar information achieved the highest accuracy and weighted F1 score, while GPT-4 with few-shot learning (using only 10 student examples) excelled in macro F1.

What Problem Does This Paper Address?

In educational settings, especially large undergraduate courses, eliciting and assessing student self-explanations is a powerful pedagogical tool. Self-explanations encourage deeper learning by prompting students to articulate their understanding, identify misconceptions, and integrate new knowledge. However, manually grading these explanations is incredibly time-consuming and impractical for instructors, leading to a reliance on less effective assessment methods like multiple-choice questions. This gap limits the widespread adoption of beneficial self-explanation activities. The paper aims to bridge this gap by investigating how LLMs can automate the assessment of student explanations in real-time, making explanation-based classroom response systems feasible.

Methods

The researchers evaluated four Transformer-based LLMs: Llama 2, GPT-3.5, GPT-4, and FLAN-T5. They explored two primary approaches:

  • Fine-tuning: For FLAN-T5, they used a supervised learning approach where the model was trained on a dataset of student explanations. They experimented with different prompt variations, including instructions only, instructions plus a rubric, instructions plus an instructor-provided exemplar response, and combinations thereof.
  • Few-Shot Learning (In-Context Learning): For Llama 2, GPT-3.5, and GPT-4, they used prompts that included task instructions, the question, the student's response, and varying amounts of additional information. This included rubric items, exemplar responses, and a limited number (ten) of other students' labeled explanations. The key innovation here is using LLMs to directly assess correctness (correct, partially correct, incorrect) without extensive retraining.

Datasets

The study utilized a dataset of 356 student explanations collected from 36 undergraduate students in an Artificial Intelligence computer science course over a 6-week period. These explanations were generated in response to 13 different questions. The student responses were manually labeled as "correct," "partially correct," or "incorrect" by two researchers, achieving substantial agreement (Cohen's Kappa of 0.702) after rubric refinement.

Key Insights and Ideas

  • Fine-tuning FLAN-T5 is a strong contender for accuracy: The results suggest that for achieving high overall accuracy and weighted F1 scores, fine-tuning an open-source model like FLAN-T5 with both rubric information and an instructor's exemplar response is highly effective. This indicates that providing structured grading criteria and a model answer can guide the model to make more accurate judgments.
  • Few-shot GPT-4 shines for nuanced classification: While FLAN-T5 excelled in overall accuracy, GPT-4, when provided with just ten labeled student examples (without rubrics or exemplars), achieved the highest macro F1 score. This suggests that LLMs can learn to discern subtle differences in correctness from a few examples, which is crucial for identifying partially correct answers.
  • Instructor exemplar responses can be tricky: Interestingly, including the instructor's exemplar response in the prompt sometimes reduced model performance for Llama 2, GPT-3.5, and GPT-4. The authors hypothesize this might be due to the exemplar being too comprehensive, leading the LLM to apply an overly strict standard. This highlights the importance of carefully crafting exemplar responses.
  • Rubric information generally helps: Across most models and approaches, incorporating rubric information improved performance, underscoring the value of explicit grading criteria.
  • Open-source vs. Proprietary Trade-offs: The paper implicitly highlights a key trade-off: FLAN-T5 (open-source) requires fine-tuning and more data but is cost-effective for large-scale deployment. GPT-4 (proprietary) offers strong few-shot performance but comes with per-token costs and less transparency.

Key Results

  • Highest Accuracy: Fine-tuned FLAN-T5 with instructions, rubric, and exemplar response achieved the highest accuracy (0.824).
  • Highest Macro F1: GPT-4 with instructions and 10 student examples achieved the highest macro F1 score (0.664).
  • Highest Weighted F1: Fine-tuned FLAN-T5 with instructions, rubric, and exemplar response achieved the highest weighted F1 score (0.798).
  • General Performance: Models consistently performed better when provided with labeled student examples.
  • Labeling Distribution: Across all questions, 73% of student explanations were labeled as correct, 22% as partially correct, and 5% as incorrect.

Limitations and Future Work

  • Comparing Fine-tuning vs. Few-Shot: A key limitation is the inherent difference in how FLAN-T5 (fine-tuned on 90% of the data) and few-shot models (using only 10 examples) were evaluated. This disparity is due to practical constraints like LLM context length limits and proprietary model costs.
  • Optimal Number of Examples: The study used a fixed number of 10 student examples for few-shot learning. Future work could investigate the optimal number of examples that balances performance and cost.
  • Exemplar Response Impact: The negative impact of instructor exemplar responses needs further investigation. Future research could explore how different characteristics of exemplar responses and rubric items affect model performance.
  • Future Directions: The authors suggest implementing the full EXPLAINIT system in a classroom to assess its impact on learning outcomes. They also propose incorporating AI for question and rubric generation and expanding the assessment to a finer-grained, concept-level analysis. Exploring the application of this approach in disciplines beyond computer science is also a promising avenue.

Summary updated

BibTeX
@inproceedings{carpenter2024assessing,
  author = {Dan Carpenter and Wookhee Min and Seung Lee and Gamze Ozogul and Xiaoying Zheng and James Lester},
  title = {{Assessing Student Explanations with Large Language Models Using Fine-Tuning and Few-Shot Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure