TL;DR
This paper investigates how to best prompt Large Language Models (LLMs) to generate helpful essay feedback for students. They explore various prompting strategies, including how to jointly perform automated essay scoring (AES) and feedback generation, finding that while joint approaches can improve scoring, the impact of scoring on feedback quality is minimal.
What Problem Does This Paper Address?
Providing individualized feedback on student essays is crucial for improving writing skills, but it's a time-consuming and labor-intensive task for educators. Automated essay feedback offers a scalable solution, but its effectiveness, especially from LLMs, is still being explored. While LLMs excel at text generation, their ability to provide helpful and specific feedback that guides students to improve remains an open question. Existing work often focuses on sentence-level grammatical errors or holistic scoring without detailed explanations. This paper aims to bridge this gap by exploring how different LLM prompting strategies can generate comprehensive, essay-level feedback and how this process can be integrated with automated essay scoring.
Methods
The authors explore various prompting strategies for LLMs to generate essay feedback. Their approach involves systematically varying three aspects of the prompts:
- Prompt Patterns: They compare a "base pattern" with "persona patterns" (e.g., "Teacher's Assistant," "Educational Researcher," "Creative Writing Mentor") to see if adopting a specific role influences the feedback quality.
- Task Instruction Types: They test different orderings and phrasings of tasks, including:
-
Scoring only
- Feedback only
- Scoring then Feedback
- Feedback then Scoring
- Scoring with Chain-of-Thought (CoT) then Feedback
- Feedback with CoT then Scoring
- Scoring with Explanation
- Explanation then Scoring This allows them to study the influence of joint task execution.
-
In-Context Learning: They compare zero-shot, one-shot, and few-shot learning approaches, providing the LLM with examples to guide its generation.
They also investigate the joint execution of Automated Essay Scoring (AES) and feedback generation, hypothesizing that these tasks might benefit from being tackled together.
Datasets
The primary dataset used in this study is the ASAP (Automated Student Assessment Prize) dataset (Hamner et al., 2012). This dataset contains approximately 12,980 essays written by students in grades 7-10, manually scored by two raters. The essays are divided into eight sets, each with a different prompt, scoring range, and rubric. The authors use this dataset for both evaluating essay scoring performance and as input for generating essay feedback. No new benchmark datasets were introduced.
Key Insights and Ideas
- Persona Prompting Matters for Scoring: Using specific personas like "Educational Researcher" or "Teacher's Assistant" improved essay scoring performance compared to a generic "base" prompt.
- Feedback-First Approaches Aid Scoring: When the LLM was instructed to generate feedback before scoring (e.g., "Feedback Scoring" or "Feedback_dCoT→Scoring"), essay scoring performance generally improved. This suggests that the process of analyzing and articulating feedback helps the model better understand the essay's quality.
- CoT Improves Scoring: Incorporating Chain-of-Thought prompting in the task instructions (e.g., "Feedback_dCoT→Scoring") also showed benefits for essay scoring.
- Feedback Quality is Key: The authors emphasize "helpfulness" as the most critical quality dimension for essay feedback. They found that generating feedback only (the "Feedback" task instruction) yielded the most helpful feedback according to both automatic LLM evaluation and manual human judgment.
- Scoring's Impact on Feedback is Limited: While joint AES and feedback generation can benefit scoring, the reverse is not as strongly supported. The impact of scoring on the helpfulness of the generated feedback was found to be low.
- LLMs Can Automate Helpfulness Evaluation: The study found a good correlation between human judgments of feedback helpfulness and scores predicted by Mistral, suggesting LLMs can be used for automatic evaluation of feedback quality.
Key Results
- Essay Scoring: The best-performing prompt pattern for scoring was the "Educational Researcher" persona, achieving a mean QWK of .509. Task instruction types that involved generating feedback first, particularly "Feedback_dCoT→Scoring" and "Explanation→Scoring," achieved the highest mean QWK of .553. One-shot learning slightly outperformed few-shot learning for scoring.
- Essay Feedback Generation (Automatic Evaluation): Both Mistral and Llama-2 models rated feedback generated by persona patterns as more helpful. The "Feedback" task instruction (feedback only) received the highest average helpfulness scores from both LLMs.
- Essay Feedback Generation (Manual Evaluation): Human annotators also found the "Feedback" task instruction to be the most helpful. Feedback was rated highly for clarity, precision, and overall helpfulness, with average scores above neutral.
- Correlation: The study found a significant positive correlation (0.61) between manually annotated overall helpfulness and automatic helpfulness scores predicted by Mistral, validating its use for automatic evaluation.
Limitations and Future Work
The authors acknowledge two main limitations:
- Data Dependence: Their feedback approaches rely on the availability of detailed rubrics and potentially textual reasoning for example essays. This might limit the transferability of their methods to datasets lacking such detailed information.
- Real-World Utilization: While their evaluation suggests the generated feedback is helpful, it remains unclear how students perceive and utilize this feedback in real-world writing scenarios.
They encourage future work to:
- Utilize their approaches for real-world essay writing support tools.
- Gather feedback directly from students on the helpfulness and usability of the generated feedback.
- Further research into how to best explain why an error is erroneous and how to make feedback more suitable for specific student demographics.
Summary updated