Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams

Rayed AlGhamdi

This paper evaluates Claude Code, an agentic deployment of the Claude Opus 4.6 model, as an automated grading assistant for handwritten university exams. While the AI achieved a remarkable 0.97 correlation with human grading, the study reveals a "reliability gradient" where the system excels at objective questions but falters on hand-drawn diagrams and exhibits a slight, systematic leniency bias.

TL;DR

Researchers tested an agentic AI system (Claude Code) on real-world, handwritten computer graphics exams featuring mixed Arabic-English text and diagrams. The AI matched human scores with high precision (r=0.97) and excellent consistency, but it struggled with conceptual depth and visual interpretation of drawings, suggesting it is best used as an instructor's "first-pass" assistant rather than a full replacement.

What Problem Does This Paper Address?

Grading handwritten STEM exams is a notorious bottleneck for educators. While Large Language Models (LLMs) have shown promise in grading typed essays, handwritten exams—especially those in technical fields—present a "boss level" challenge. They often involve:

  1. Multimodal inputs: A mix of text, mathematical notation, and hand-drawn diagrams.
  2. Multilingualism: Students often switch between languages (e.g., Arabic and English) for technical terms.
  3. Non-deterministic corrections: Students frequently cross out answers or annotate diagrams in ways that confuse standard OCR.

The author sought to determine if an agentic AI—one that can autonomously manage files, use tools, and follow complex multi-file rubrics—could handle these complexities with enough reliability to be used in a real university setting.

Methods

The study utilized Claude Code (version 2.1.117), which differs from standard chat interfaces by operating in a local development environment with direct file access.

The grading pipeline was architected as an "orchestrator" session that read three structured Markdown files:

  • Grading Rules: Behavioral guidelines (e.g., how to handle Arabic-English mix).
  • Marking Scheme: The actual answer key and partial credit logic.
  • Output Schema: The exact format for grade reports and "flags" for human review.

The orchestrator then dispatched ten parallel sub-agents to grade batches of student papers. To measure reliability, the author used a "blind" protocol where human grading was completed before inspecting AI results. The study analyzed two disjoint samples: an "agreement" sample (20 papers) to compare AI vs. Human, and a "consistency" sample (15 papers) to test if the AI would give the same grade to the same paper one week later.

Datasets

The study used a primary dataset of 60 handwritten mid-term exams from a 300-level Computer Graphics course at King Abdulaziz University.

  • Format: Scanned PDFs of handwritten answers.
  • Content: 7 questions covering Bloom’s Taxonomy levels L1 (Recall) through L5 (Evaluate).
  • Language: Mixed Arabic and English.

The paper positions itself in a unique niche of the research landscape, as shown in Figure 1, by tackling high-complexity response modalities (handwritten mixed-format) within an agentic environment.

Figure 1: Peer-reviewed LLM grading studies by deployment condition and response modality

Key Insights and Ideas

  • The "Reliability Gradient": AI performance isn't binary. There is a clear drop-off in reliability as questions move from objective (MCQs) to comprehension (written) to visual (drawings).
  • "Reliably Wrong": The AI showed excellent test-retest consistency (ICC = 0.96), meaning it rarely changed its mind. However, it was consistently more lenient than the human grader by about 0.68 marks. This highlights that stability does not equal accuracy.
  • Surface vs. Depth: A recurring failure mode was "keyword matching without conceptual depth." The AI would award marks if a student used the right vocabulary (e.g., "smooth," "rough"), even if the student failed to explain the underlying mechanism.
  • Selection-State Ambiguity: In a uniquely "handwritten" problem, the AI sometimes struggled to tell which circled answer was the "final" one if a student had crossed out a previous choice.

Key Results

  • Overall Agreement: Strong correlation (r = 0.97) and a low Mean Absolute Error (MAE = 0.85 marks on a 20-mark scale).
  • Systematic Bias: The AI exhibited a statistically significant positive bias of +0.68 marks, tending to be slightly more generous than the human expert.
  • Feedback Accuracy: The AI-generated feedback was remarkably accurate, with 92.8% of diagnostic comments (explaining why marks were lost) being substantively correct.
  • The Drawing Outlier: Drawing questions were the clear weak link, with an exact agreement rate of only 55% and a much lower consistency score (ICC = 0.52).

Limitations and Future Work

The authors acknowledge several constraints:

  • Sample Size: The study relied on a relatively small cohort (60 students) from a single institution.
  • Single Rater: Human grading was performed only by the course instructor. A multi-rater human consensus would provide a more robust benchmark.
  • Validity vs. Reliability: This paper focuses strictly on reliability (consistency and agreement). It does not address validity—whether the AI is actually measuring the intended learning outcomes or if students find the AI-graded experience "fair."

Future research directions include testing newer model iterations (like Claude 3.7), exploring specialized vision-language pipelines to improve diagram grading, and conducting cross-institutional replications to see if these results hold across different subjects and languages.

Summary updated

Abstract

This study examines the reliability of Claude Code, an agentic deployment of the Claude Opus 4.6 large language model, as a grading system for handwritten university exams. Four research questions addressed agreement with expert human grading on total scores (RQ1), agreement across five question types from multiple choice to evaluative judgement (RQ2), consistency across two independent grading sessions one week apart (RQ3), and the accuracy of AI-generated feedback (RQ4). Claude Code graded the 20-mark mid-term papers of 60 consenting students in a Saudi computer graphics course in a single agentic session, guided by three rubric files. Reliability was estimated on disjoint samples: 20 papers for RQ1, RQ2 and RQ4, and 15 papers for RQ3. Analyses used Pearson correlation, mean absolute error, Bland-Altman analysis, weighted Cohen kappa, intraclass correlation, and a four-category coding scheme separating diagnostic from confirmation feedback. The author, as the sole subject-matter expert for the course, provided the human benchmark, so the design assesses agreement with a domain-expert reference rather than inter-rater reliability. AI grading agreed strongly with human grading on totals (r = 0.97; MAE = 0.85 marks) with a small but significant AI-positive bias of 0.68 marks, showed excellent test-retest consistency (ICC = 0.96), and produced accurate feedback (92.8% diagnostic; 97.8% confirmation). Drawing items were the outlier on three of four measures. Three failure modes recurred: keyword matching without conceptual depth, selection-state ambiguity on corrected multiple-choice answers, and region-of-attention drift on hand-drawn diagrams. The findings suggest that AI grading at this capability level can assist instructors on objective and short-form items under structural safeguards but is not yet ready to replace instructor judgement at boundary cases or on drawing items.
BibTeX
@misc{alghamdi2026reliability,
  author = {Rayed AlGhamdi},
  title = {{Reliability of Claude Code as an Agentic AI Grading System for Handwritten University Exams}},
  year = {2026},
  eprint = {2611.13217},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13217},
}
Figure