When AI explains in natural language: Unveiling the impact of generative AI explanations on educators’ grading and feedback practices

Yuheng Li, Zirui Shan, Mladen Raković, Quanlong Guan, Dragan Gašević, Guanliang ChenEducation and Information Technologies 2025 · OpenReview

Introduction

In the evolving landscape of education, the assessment of open-ended student responses remains a significant challenge. Grading is not only time-consuming and cognitively demanding but also susceptible to subjective bias. While Automated Assessment Systems (AAS) have been proposed as a solution, their adoption is often hindered by concerns regarding accuracy and the "black box" nature of their decision-making. This paper explores a "human-in-the-loop" approach, investigating how Generative AI (GenAI) can provide natural-language explanations to support educators in their grading and feedback practices.

The Problem: The Interpretability Gap

Traditional Explainable AI (XAI) in education often relies on feature-based explanations, such as highlighting "important words" that contributed to a score. However, these methods frequently fail to bridge the interpretability gap. For an educator, seeing a set of highlighted words may actually increase cognitive load, as they must still deduce the underlying rationale. The authors hypothesize that natural-language explanations, powered by GenAI, could reduce this burden by providing clear, step-by-step rationales that align more closely with human pedagogical reasoning.

Methodology

To test this hypothesis, the researchers conducted a controlled study involving 60 experienced educators. The study utilized the ASAP-SAS (Automated Student Assessment Prize - Short Answer Scoring) dataset, focusing on Grade 10 English summarization and science reasoning tasks.

The experiment was structured into four phases:

  1. Pre-Training: Participants were familiarized with rubrics and feedback guidelines.
  2. Pre-Evaluation: A baseline phase where participants graded 12 answers without AI assistance.
  3. Intervention (Exp-1): Participants were randomly assigned to one of three groups:
  4. Control Group: No AI support.
  5. Traditional AI Group: Received "important-word highlights" generated by a BERT-based model.
  6. GenAI Group: Received natural-language rationales generated by GPT-4 using chain-of-thought reasoning.
  7. Retention (Exp-2): One week later, participants graded a new set of answers without any AI support to measure lasting impacts on their assessment skills.

Mixed-effect regression analyses were employed to evaluate the impact of these insights on grading accuracy and feedback quality, accounting for individual grader variability and task difficulty.

Key Findings

1. Significant Improvement in Feedback Quality

The most striking result was the impact on the quality of feedback provided to students. Educators in the GenAI group produced significantly higher-quality feedback compared to those without support ($\beta = 0.190, p = 0.010$). These natural-language insights helped educators identify specific merits and shortcomings in student work, leading to more learner-centered feedback that emphasized student agency and sense-making. In contrast, the traditional "word highlighting" approach had a negligible effect on feedback quality.

2. Potential for Enhanced Grading Accuracy

While the study did not find a statistically significant improvement in grading accuracy for the GenAI group ($\beta = 0.556, p = 0.093$), the trend was notably more positive than that of the word-highlighting group ($\beta = -0.060, p = 0.848$). The authors suggest that the relatively straightforward nature of the tasks for experienced educators may have limited the room for measurable accuracy gains, but the potential for GenAI to support accuracy in more complex tasks remains promising.

3. Higher User Satisfaction and Adoption Willingness

Educators reported significantly greater satisfaction and a higher willingness to adopt natural-language insights in their daily practice compared to word highlights ($p < 0.05$). The word-highlighting method was often perceived as less comprehensible, whereas the GenAI rationales were seen as helpful auxiliary information.

4. The "Training Effect"

A surprising and critical finding was the evidence of a "training effect." Educators who were previously exposed to GenAI insights showed improved assessment practices in the retention phase, even after the AI support was removed. This suggests that interacting with high-quality AI explanations can help educators internalize better assessment methodologies, effectively serving as a form of on-the-job professional development.

Discussion and Future Directions

The study demonstrates that GenAI-powered natural-language explanations can effectively support educators by reducing the cognitive effort required to interpret AI-generated scores. By moving beyond simple data visualization to conversational rationales, GenAI helps educators focus on providing meaningful feedback.

However, several questions remain. This study focused on instances where the AI was correct. Future research must investigate how educators respond to incorrect AI insights to mitigate the risk of "metacognitive laziness" or over-reliance. Additionally, while the natural-language insights improved quality, they did not necessarily save time, as educators spent significant effort digesting the explanations.

As AI continues to integrate into educational workflows, the goal should not be to replace human judgment but to augment it. This research highlights a path toward collaborative human-AI systems that enhance both the efficiency and the pedagogical depth of student assessment.

Summary updated

Abstract

Grading and feedback provision for students’ open-ended responses are time-consuming and cognitively demanding. Despite the advocacy to leverage Artificial Intelligence (AI) to automate assessment, concerns persist regarding inaccurate AI assessment on student learning and the potential detrimental effects on educators’ assessment practices due to reliance. One alternative is to leverage AI-powered assessment insights as auxiliary information to support educators’ assessment instead of solely entrusting AI for assessment. However, insights published in existing literature typically required excessive cognitive efforts from human graders to interpret and make use of. Generative AI (GenAI) technologies’ capability To produce natural-language assessment explanations could potentially tackle this challenge. To empirically examine the efficacy of such natural-language insights while accounting for concerns in over-reliance on AI, we invited 60 human graders from diverse backgrounds to participate in multiple phases of assessment for secondary students’ short-answer responses. Participants were assigned to one of the three conditions: no AI-powered assessment support; important-word highlights from AI-powered graders as support; and natural-language grading explanations from GenAI-powered graders as support. Mixed-effect regression analyses were adopted to examine the impacts of different AI-powered assessment insights on human assessment practices both in current assessment with AI-powered insights presented and in later assessment without AI-powered insights. Our findings indicate that GenAI-enabled natural-language insights significantly improved educators’ feedback quality compared to educators without AI support (\(\beta =0.190\), \(p=0.010\)). Important-word highlights from traditional AI graders had negligible impact on educators’ feedback quality. Although significant improvements in grading accuracy were not observed, natural-language insights showed greater potential to enhance grading accuracy (\(\beta =0.556\), \(p=0.093\)) than important-word highlights (\(\beta =-0.060\), \(p=0.848\)). Furthermore, educators reported significantly greater satisfaction and willingness to adopt the natural-language insights in practice compared to the important-word insights (\(p<0.05\), \(r \in [0.352, 0.490]\)). Finally, prior exposure to AI-powered insights to some extent fostered educators’ more effective assessment practices in subsequent assessment activities without AI support, though further longitudinal research is needed to establish empirical significance.
BibTeX
@inproceedings{li2025when,
  author = {Yuheng Li and Zirui Shan and Mladen Raković and Quanlong Guan and Dragan Gašević and Guanliang Chen},
  title = {{When AI explains in natural language: Unveiling the impact of generative AI explanations on educators’ grading and feedback practices}},
  booktitle = {Education and Information Technologies},
  year = {2025},
  url = {https://openreview.net/forum?id=57e2hqnP9T},
}
Figure