Abstract
Educational feedback is a cornerstone of effective learning, yet scaling high-quality, personalized feedback remains a significant challenge in higher education. This study investigates the feasibility of using ChatGPT to provide students with feedback on open-ended writing assignments. By examining readability, agreement with human instructors, and pedagogical depth using the Hattie and Timperley framework, the researchers evaluate whether Large Language Models (LLMs) can advance Automated Feedback Systems (AFS). The results indicate that while ChatGPT generates more detailed and readable feedback than human instructors and excels at suggesting learning strategies, it exhibits a notable "positivity bias" that may limit its evaluative accuracy.
Introduction
Quality feedback is one of the most effective tools for student success, but it is notoriously difficult to scale. In large university cohorts, instructors often face time constraints that lead to feedback being either delayed or too brief to be constructive. Traditional Automated Feedback Systems (AFS) often rely on rigid, rule-based logic that struggles with the nuances of open-ended tasks like essay writing. This research explores whether ChatGPT can bridge this gap by generating natural-sounding, context-specific, and pedagogically useful feedback for complex assignments.
Methodology
The study utilized a dataset of 103 anonymized student project proposals from a postgraduate data science course at an Australian university. ChatGPT was prompted to provide feedback based on a five-point marking rubric covering: Goal, Topic, Benefit, Novelty, and Clarity. The evaluation was conducted through three lenses:
- Readability: Experts scored feedback on a 5-point scale for fluency and coherence.
- Agreement (Polarity): Experts categorized the "polarity" (Positive, Negative, or None) of the feedback to determine alignment with human instructor assessments.
- Pedagogical Effectiveness: Feedback was categorized using the Hattie and Timperley (2007) framework into four levels: Task (corrective), Process (strategies), Self-regulation (monitoring), and Self (personal evaluation).
Results and Discussion
Readability and Detail
ChatGPT-generated feedback was rated as significantly more readable and detailed than human feedback, achieving a mean score of 3.76 compared to the instructors' 3.21 (p < 0.001). While instructors, constrained by time, often provided terse expressions, ChatGPT offered comprehensive summaries and explanations justifying its comments.
The Politeness Trap: Polarity and Agreement
A critical finding was ChatGPT’s strong tendency toward positive reinforcement, often at the expense of critical accuracy. While agreement was high for the "Topic" aspect (Precision: 0.84, Recall: 0.93), it was remarkably low for the "Goal" aspect.
As shown in Table 4, for the "Goal" aspect, the instructor gave positive feedback to only 5 reports, whereas ChatGPT gave positive feedback to 95. This suggests that without specific "golden" examples or few-shot prompting, the model acts more as a cheerleader than a critical evaluator.
Pedagogical Effectiveness: Feedback Levels
One of the most promising results was ChatGPT’s ability to provide process-level feedback. Unlike many automated systems that only provide corrective (Task-level) feedback, ChatGPT suggested strategies for completing tasks in 55% of instances, compared to 80% for instructors.
However, the AI completely failed to provide "Self-regulation" or "Self-level" feedback, which involve encouraging students to monitor their own work or providing personal encouragement. While instructors provided these levels occasionally (11% and 24% respectively), the AI remained strictly focused on the document content itself.
Conclusion
While ChatGPT is not yet ready to replace human instructors due to its lack of evaluative grounding and positivity bias, it shows great potential as a "drafting" tool. By generating detailed, readable feedback that instructors can then review and refine, LLMs could significantly reduce the manual workload of educators while maintaining pedagogical depth. Future work should explore "few-shot" prompting to improve assessment accuracy and mitigate the model's inherent politeness.
Summary updated