TL;DR
This study investigates whether ChatGPT can effectively replace or augment human instructors in providing feedback on open-ended writing assignments. While ChatGPT produces significantly more readable and detailed feedback than busy instructors and excels at suggesting learning strategies (process-level feedback), it currently suffers from a "positivity bias," often praising student work that instructors marked as needing improvement.
What Problem Does This Paper Address?
Quality feedback is one of the most effective tools for student success, yet it is notoriously difficult to scale. In higher education, instructors often manage large cohorts, leading to feedback that is either delayed or too brief to be truly constructive.
While Automated Feedback Systems (AFS) have existed for years, they typically rely on rigid, expert-defined rules that fail when applied to open-ended tasks like essay writing or project proposals. The authors explore whether Large Language Models (LLMs), specifically ChatGPT, can bridge this gap by generating natural-sounding, context-specific, and pedagogically useful feedback for complex assignments.
Methods
The researchers conducted a case study using 103 student project proposals from a postgraduate data science course. They prompted ChatGPT to provide feedback based on a specific five-point marking rubric (Goal, Topic, Benefit, Novelty, and Clarity).
To evaluate the AI's performance, they employed three distinct lenses:
- Readability: Three experts scored the feedback on a 5-point scale for fluency and coherence.
- Agreement (Polarity): Two experts categorized the "polarity" (Positive, Negative, or None) of the feedback for each rubric aspect to see if ChatGPT’s assessment aligned with the human instructor's.
- Pedagogical Effectiveness: The authors used the Hattie and Timperley (2007) framework, which categorizes feedback into four levels: Task (corrective), Process (strategies), Self-regulation (monitoring), and Self (personal evaluation).
Datasets
The study utilized a real-world dataset of 103 anonymized student proposal reports and their corresponding instructor-generated feedback from an Australian university. The proposals focused on data science applications in business scenarios. No new public benchmark was introduced, as the data was kept private for ethics and privacy reasons.
Key Insights and Ideas
- The "Terse Instructor" vs. "Verbose AI": Interestingly, ChatGPT-generated feedback was rated as significantly more readable and detailed than human feedback. Instructors, constrained by time, often provide "simple expressions," whereas ChatGPT provides summaries and explanations of why a certain grade or comment was given.
- The Politeness Trap: ChatGPT exhibited a strong tendency toward positive reinforcement. In many cases where an instructor provided negative feedback (e.g., regarding the "Goal" of a project), ChatGPT provided positive feedback. This suggests that without specific "golden" examples or few-shot prompting, the model acts more like a cheerleader than a critical evaluator.
- Beyond "Right or Wrong": One of the most surprising findings was ChatGPT’s ability to provide Process-level feedback. While many automated systems just tell a student if they are wrong (Task-level), ChatGPT was able to suggest strategies for completing the task in over 50% of the cases, a key component for developing long-term learning skills.
- The Missing "Self": ChatGPT completely failed to provide "Self-regulation" or "Self-level" feedback. These levels involve encouraging students to monitor their own work or providing personal encouragement (e.g., "You are a great student"). Instructors did this occasionally, but the AI remained strictly focused on the document itself.
Key Results
- Readability: ChatGPT outperformed instructors with a mean score of 3.76 vs. 3.21 (p < 0.001).
- Assessment Agreement: Agreement was highest on the "Topic" aspect (Precision: 0.84, Recall: 0.93). However, it was dismal on "Goal" (Precision: 0.04), where ChatGPT gave positive feedback to 95 reports, while the instructor only gave positive feedback to 5.
- Feedback Levels: Both ChatGPT and instructors provided Task-level feedback 100% of the time. However, ChatGPT provided Process-level feedback in 55% of instances, compared to 80% for instructors.
Limitations and Future Work
The authors acknowledge several critical hurdles:
- Lack of Grounding: The study measured "polarity" agreement but didn't strictly test if the AI's feedback was factually accurate relative to the specific student's submission.
- Prompt Engineering: The model was used in an "unsupervised" way with a simple prompt. The authors suggest that "few-shot" prompting—providing the AI with examples of high-quality human feedback—might fix the positivity bias and improve assessment accuracy.
- Human-in-the-loop: The researchers conclude that while ChatGPT isn't ready to replace instructors, it could serve as a powerful "drafting" tool, generating detailed feedback that an instructor then reviews and corrects, significantly reducing the manual workload.
Summary updated