The application of GPT-4 in grading design university students' assignment and providing feedback: An exploratory study

Qian Huang, Thijs Willems, King Wang PoonarXiv 2024 · DOI · arXiv

Introduction

Grading in design education is notoriously subjective. Unlike disciplines with clear-cut answers, design problems are open-ended, often leading to significant discrepancies between human raters. For instance, an instructor with an engineering background might prioritize functionality, while an architecture instructor might focus on spatial aesthetics. This exploratory study investigates whether GPT-4 can bridge this gap by providing consistent, rubric-based grading and personalized feedback that synthesizes multiple disciplinary perspectives.

Methodology

The researchers employed a Design-Based Research (DBR) approach, an iterative process of testing and refinement. The study evaluated GPT-4 across three primary "comparison lines" to assess its reliability as a grading tool.

The iterative experimental design across three parts: inter-reliability without role-play, with role-play, and intra-reliability.

As shown in Figure 1, the process involved:

  1. Inter-reliability without role-play: Testing three iterations from a General GPT to a Custom GPT enhanced with specific rubrics and "gold standard" examples.
  2. Inter-reliability with role-play: Prompting the AI to adopt specific professional personas (e.g., architecture vs. engineering instructor).
  3. Intra-reliability: Testing the model’s consistency by grading the same assignments at three different times over several weeks.

The primary metric used was the Intraclass Correlation Coefficient (ICC), which measures the level of agreement between different raters.

Key Results

The study first established a baseline by comparing human instructors. The initial agreement between the Architect and Engineer instructors was remarkably low, highlighting the challenge of subjective assessment in design.

Initial Intraclass Correlation Coefficient (ICC) comparisons showing poor reliability between human instructors and early GPT iterations.

While early comparisons showed poor reliability (Figure 3), the performance improved significantly through iteration. In the final stage (Custom GPT + Rubrics + Example), the agreement between the AI and the average of human instructors reached an ICC of 0.7285. In educational psychometrics, a score above 0.70 is considered "good" and acceptable for assessment purposes. Furthermore, GPT-4 demonstrated high intra-rater reliability (ICC up to 0.78), providing a stable baseline that human graders—often subject to fatigue—may lack.

Discussion: The "Good Example" and Disciplinary Mediation

A critical finding was that providing a "gold standard" example was the most effective way to improve AI performance. Concrete anchors for "excellence" proved more valuable than abstract rubric descriptions alone.

Additionally, GPT-4 acted as a disciplinary mediator. It successfully synthesized feedback from both architectural and engineering perspectives, helping students navigate conflicting advice. This ability to provide 24/7 personalized feedback and draw comparisons to renowned designers offers a level of support that human instructors often lack the time to provide individually.

Conclusion and Limitations

While promising, the study notes limitations including a small sample size (10 students) and concerns regarding data privacy and inherent AI bias. The authors emphasize that GPT-4 should be viewed as a "co-worker" rather than a replacement for human educators. Human domain knowledge remains essential for final oversight and identifying creative nuances that AI might overlook. Future work will focus on using AI to help human raters "calibrate" their own grading by identifying personal biases relative to rubric-anchored AI benchmarks.

Summary updated

BibTeX
@misc{huang2024application,
  author = {Qian Huang and Thijs Willems and King Wang Poon},
  title = {{The application of GPT-4 in grading design university students' assignment and providing feedback: An exploratory study}},
  year = {2024},
  eprint = {2409.17698},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.17698},
}
Figure