This paper evaluates Claude Code, an agentic deployment of the Claude Opus 4.6 model, as an automated grading assistant for handwritten university exams. While the AI achieved a remarkable 0.97 correlation with human grading, the study reveals a "reliability gradient" where the system excels at objective questions but falters on hand-drawn diagrams and exhibits a slight, systematic leniency bias.
TL;DR
Researchers tested an agentic AI system (Claude Code) on real-world, handwritten computer graphics exams featuring mixed Arabic-English text and diagrams. The AI matched human scores with high precision (r=0.97) and excellent consistency, but it struggled with conceptual depth and visual interpretation of drawings, suggesting it is best used as an instructor's "first-pass" assistant rather than a full replacement.
What Problem Does This Paper Address?
Grading handwritten STEM exams is a notorious bottleneck for educators. While Large Language Models (LLMs) have shown promise in grading typed essays, handwritten exams—especially those in technical fields—present a "boss level" challenge. They often involve:
- Multimodal inputs: A mix of text, mathematical notation, and hand-drawn diagrams.
- Multilingualism: Students often switch between languages (e.g., Arabic and English) for technical terms.
- Non-deterministic corrections: Students frequently cross out answers or annotate diagrams in ways that confuse standard OCR.
The author sought to determine if an agentic AI—one that can autonomously manage files, use tools, and follow complex multi-file rubrics—could handle these complexities with enough reliability to be used in a real university setting.
Methods
The study utilized Claude Code (version 2.1.117), which differs from standard chat interfaces by operating in a local development environment with direct file access.
The grading pipeline was architected as an "orchestrator" session that read three structured Markdown files:
- Grading Rules: Behavioral guidelines (e.g., how to handle Arabic-English mix).
- Marking Scheme: The actual answer key and partial credit logic.
- Output Schema: The exact format for grade reports and "flags" for human review.
The orchestrator then dispatched ten parallel sub-agents to grade batches of student papers. To measure reliability, the author used a "blind" protocol where human grading was completed before inspecting AI results. The study analyzed two disjoint samples: an "agreement" sample (20 papers) to compare AI vs. Human, and a "consistency" sample (15 papers) to test if the AI would give the same grade to the same paper one week later.
Datasets
The study used a primary dataset of 60 handwritten mid-term exams from a 300-level Computer Graphics course at King Abdulaziz University.
- Format: Scanned PDFs of handwritten answers.
- Content: 7 questions covering Bloom’s Taxonomy levels L1 (Recall) through L5 (Evaluate).
- Language: Mixed Arabic and English.
The paper positions itself in a unique niche of the research landscape, as shown in Figure 1, by tackling high-complexity response modalities (handwritten mixed-format) within an agentic environment.
Key Insights and Ideas
- The "Reliability Gradient": AI performance isn't binary. There is a clear drop-off in reliability as questions move from objective (MCQs) to comprehension (written) to visual (drawings).
- "Reliably Wrong": The AI showed excellent test-retest consistency (ICC = 0.96), meaning it rarely changed its mind. However, it was consistently more lenient than the human grader by about 0.68 marks. This highlights that stability does not equal accuracy.
- Surface vs. Depth: A recurring failure mode was "keyword matching without conceptual depth." The AI would award marks if a student used the right vocabulary (e.g., "smooth," "rough"), even if the student failed to explain the underlying mechanism.
- Selection-State Ambiguity: In a uniquely "handwritten" problem, the AI sometimes struggled to tell which circled answer was the "final" one if a student had crossed out a previous choice.
Key Results
- Overall Agreement: Strong correlation (r = 0.97) and a low Mean Absolute Error (MAE = 0.85 marks on a 20-mark scale).
- Systematic Bias: The AI exhibited a statistically significant positive bias of +0.68 marks, tending to be slightly more generous than the human expert.
- Feedback Accuracy: The AI-generated feedback was remarkably accurate, with 92.8% of diagnostic comments (explaining why marks were lost) being substantively correct.
- The Drawing Outlier: Drawing questions were the clear weak link, with an exact agreement rate of only 55% and a much lower consistency score (ICC = 0.52).
Limitations and Future Work
The authors acknowledge several constraints:
- Sample Size: The study relied on a relatively small cohort (60 students) from a single institution.
- Single Rater: Human grading was performed only by the course instructor. A multi-rater human consensus would provide a more robust benchmark.
- Validity vs. Reliability: This paper focuses strictly on reliability (consistency and agreement). It does not address validity—whether the AI is actually measuring the intended learning outcomes or if students find the AI-graded experience "fair."
Future research directions include testing newer model iterations (like Claude 3.7), exploring specialized vision-language pipelines to improve diagram grading, and conducting cross-institutional replications to see if these results hold across different subjects and languages.
Summary updated