TL;DR
This paper presents an open-source, locally deployable pipeline that uses Small Language Models (SLMs) to automate the rubric-based grading of complex student reports. By combining text-based SLMs with vision-language models for image analysis, the authors demonstrate that compact models running on consumer-grade hardware can achieve up to 97% accuracy, rivaling much larger models while ensuring student data privacy.
What Problem Does This Paper Address?
Grading student reports is a significant bottleneck for instructors, especially in technical fields like cybersecurity where reports often include "rich-text" elements like screenshots and specific formatting. While Large Language Models (LLMs) offer a solution for automated grading, they present three major hurdles:
- Privacy: Sending student data to cloud providers (like OpenAI) can violate regulations like GDPR or FERPA.
- Cost and Resources: High-end LLMs are expensive and require massive computational power that many educational institutions lack.
- Multimodality: Most automated grading research focuses on plain text, ignoring the visual evidence (e.g., screenshots of a phishing attack) often required in technical documentation.
The authors tackle these issues by investigating whether "small" models (under 10B parameters) can be run locally on modest hardware to provide accurate, rubric-aligned feedback.
Methods
The researchers developed a multi-stage pipeline designed to run on-device using the Ollama framework. The process follows these steps:
- Data Extraction: A Python script uses
PyPDF2andPillowto separate text from embedded images in student PDF submissions. - Visual Interpretation: Because standard text models cannot "see," a Vision-Language Model (llava:7b) analyzes the extracted images and generates textual descriptions based on the grading rubric (e.g., "This image shows a webmail login form").
- Rubric-Aligned Prompting: The combined text and image descriptions are fed into an SLM (such as Llama 3: 8B or Mistral 7B). The prompt includes a specific persona (instructor), the context of the assignment, the strict rubric criteria, and an exemplary report.
- Structured Output: The models are constrained to output scores and feedback in a structured JSON format, making it easy to integrate into learning management systems.
Datasets
The study utilized a real-world dataset from an undergraduate "Cybersecurity in an Organization" course:
- Participants: 105 students (organized into 35 teams).
- Documents: Two types of reports generated during a simulated incident response exercise:
- Executive Summary (ES): A text-only summary for management.
- Security Advisory (SA): A rich-text PDF intended for end-users, containing both text and screenshots of phishing emails/websites.
- Ground Truth: Manual grades provided by 5–6 instructors and verified by the course guarantor.
Key Insights and Ideas
- SLMs Punch Above Their Weight: Small models (7B-8B parameters) frequently matched or outperformed a massive 120B parameter baseline model. This suggests that for specific, rubric-constrained tasks, "bigger" isn't always "better."
- Evidence vs. Context: Models were highly accurate (90%+) at identifying specific pieces of evidence, such as the presence of a phishing sender's email address. However, they struggled with "context-heavy" criteria, such as evaluating the overall logical structure of a document.
- Hardware Accessibility: The pipeline was tested on various setups, including a laptop GPU and an older 2018-era desktop GPU (RTX 2080Ti). All configurations successfully ran the text-only evaluations, proving that AI-assisted grading doesn't require a supercomputer.
- Strict Rubrics are Key: The authors found that moving away from open-ended "essay" grading toward binary (0/1) or narrow ordinal (0-2) scales significantly improved model reliability and reduced subjectivity.
Key Results
The researchers compared five models across four hardware configurations.
- Accuracy: The models achieved up to 97% accuracy for text-only reports (Executive Summaries) and up to 94% accuracy for rich-text reports (Security Advisories).
- Speed: Evaluating a single student report took between 30 and 90 seconds depending on the model and hardware. While the reasoning-heavy Qwen: 1.7B was the slowest, Gemma: 4B and Llama 3: 8B offered an excellent balance of speed and precision.
- Consistency: Accuracy remained stable across multiple runs (as shown by narrow confidence intervals in Figure 5), indicating that the models are reliable enough for a "first-pass" triage of student work.
Limitations and Future Work
- The "Final Call": The authors emphasize that this system is a support tool, not a replacement. Human oversight is still required, especially for criteria where accuracy fell below 55% (like document sectioning).
- Contextual Blind Spots: Models occasionally failed to recognize the broader context of the assignment, even when it was included in the prompt.
- Generalizability: The current study is limited to one cybersecurity course. Future work will explore how this workflow adapts to other disciplines and more complex, multi-modal evidence.
- Human Error: The study assumes human scores are the "ground truth," though instructors can also make mistakes. Future iterations might look at cases where the model's "disagreement" actually highlights a human grading error.
Summary updated