Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT

Wei Dai, Jionghao Lin, Flora Jin, Tongguang Li, Yi‐Shan Tsai, Dragan Gašević, Guanliang Chen

Abstract

Educational feedback is a cornerstone of effective learning, yet scaling high-quality, personalized feedback remains a significant challenge in higher education. This study investigates the feasibility of using ChatGPT to provide students with feedback on open-ended writing assignments. By examining readability, agreement with human instructors, and pedagogical depth using the Hattie and Timperley framework, the researchers evaluate whether Large Language Models (LLMs) can advance Automated Feedback Systems (AFS). The results indicate that while ChatGPT generates more detailed and readable feedback than human instructors and excels at suggesting learning strategies, it exhibits a notable "positivity bias" that may limit its evaluative accuracy.

Introduction

Quality feedback is one of the most effective tools for student success, but it is notoriously difficult to scale. In large university cohorts, instructors often face time constraints that lead to feedback being either delayed or too brief to be constructive. Traditional Automated Feedback Systems (AFS) often rely on rigid, rule-based logic that struggles with the nuances of open-ended tasks like essay writing. This research explores whether ChatGPT can bridge this gap by generating natural-sounding, context-specific, and pedagogically useful feedback for complex assignments.

Methodology

The study utilized a dataset of 103 anonymized student project proposals from a postgraduate data science course at an Australian university. ChatGPT was prompted to provide feedback based on a five-point marking rubric covering: Goal, Topic, Benefit, Novelty, and Clarity. The evaluation was conducted through three lenses:

  1. Readability: Experts scored feedback on a 5-point scale for fluency and coherence.
  2. Agreement (Polarity): Experts categorized the "polarity" (Positive, Negative, or None) of the feedback to determine alignment with human instructor assessments.
  3. Pedagogical Effectiveness: Feedback was categorized using the Hattie and Timperley (2007) framework into four levels: Task (corrective), Process (strategies), Self-regulation (monitoring), and Self (personal evaluation).

Results and Discussion

Readability and Detail

ChatGPT-generated feedback was rated as significantly more readable and detailed than human feedback, achieving a mean score of 3.76 compared to the instructors' 3.21 (p < 0.001). While instructors, constrained by time, often provided terse expressions, ChatGPT offered comprehensive summaries and explanations justifying its comments.

The Politeness Trap: Polarity and Agreement

A critical finding was ChatGPT’s strong tendency toward positive reinforcement, often at the expense of critical accuracy. While agreement was high for the "Topic" aspect (Precision: 0.84, Recall: 0.93), it was remarkably low for the "Goal" aspect.

Table 4: Polarity and agreement metrics across rubric aspects, highlighting the discrepancy in assessment between the instructor and ChatGPT.

As shown in Table 4, for the "Goal" aspect, the instructor gave positive feedback to only 5 reports, whereas ChatGPT gave positive feedback to 95. This suggests that without specific "golden" examples or few-shot prompting, the model acts more as a cheerleader than a critical evaluator.

Pedagogical Effectiveness: Feedback Levels

One of the most promising results was ChatGPT’s ability to provide process-level feedback. Unlike many automated systems that only provide corrective (Task-level) feedback, ChatGPT suggested strategies for completing tasks in 55% of instances, compared to 80% for instructors.

Table 6: Frequency of feedback levels according to the Hattie and Timperley framework.

However, the AI completely failed to provide "Self-regulation" or "Self-level" feedback, which involve encouraging students to monitor their own work or providing personal encouragement. While instructors provided these levels occasionally (11% and 24% respectively), the AI remained strictly focused on the document content itself.

Conclusion

While ChatGPT is not yet ready to replace human instructors due to its lack of evaluative grounding and positivity bias, it shows great potential as a "drafting" tool. By generating detailed, readable feedback that instructors can then review and refine, LLMs could significantly reduce the manual workload of educators while maintaining pedagogical depth. Future work should explore "few-shot" prompting to improve assessment accuracy and mitigate the model's inherent politeness.

Summary updated

Abstract

Educational feedback has been widely acknowledged as an effective approach to improving student learning. However, scaling effective practices can be laborious and costly, which motivated researchers to work on automated feedback systems (AFS). Inspired by the recent advancements in the pre-trained language models (e.g., ChatGPT), we posit that such models might advance the existing knowledge of textual feedback generation in AFS because of their capability to offer natural-sounding and detailed responses. Therefore, we aimed to investigate the feasibility of using ChatGPT to provide students with feedback to help them learn better. Specifically, we first examined the readability of ChatGPT-generated feedback. Then, we measured the agreement between ChatGPT and the instructor when assessing students' assignments according to the marking rubric. Finally, we used a well-known theoretical feedback framework to further investigate the effectiveness of the feedback generated by ChatGPT. Our results show that i) ChatGPT is capable of generating more detailed feedback that fluently and coherently summarizes students' performance than human instructors; ii) ChatGPT achieved high agreement with the instructor when assessing the topic of students' assignments; and iii) ChatGPT could provide feedback on the process of students completing the task, which benefits students developing learning skills.
BibTeX
@inproceedings{dai2023can,
  author = {Wei Dai and Jionghao Lin and Flora Jin and Tongguang Li and Yi‐Shan Tsai and Dragan Gašević and Guanliang Chen},
  title = {{Can Large Language Models Provide Feedback to Students? A Case Study on ChatGPT}},
  booktitle = {ICALT},
  year = {2023},
  url = {https://openreview.net/forum?id=3EMwueko4S},
}
Figure