Abstract
Automating the evaluation of open-ended science assessments remains a challenge due to the non-canonical nature of student responses. This paper introduces a human-in-the-loop framework that leverages GPT-4, Chain-of-Thought (CoT) prompting, and Active Learning (AL) to score and explain middle school science responses. Our results demonstrate that this approach achieves high alignment with human graders (QWK ≥ 0.8) and provides pedagogically sound reasoning, offering a scalable solution for formative feedback in K-12 STEM education.
Introduction
Formative assessment—the frequent, low-stakes checking of student understanding—is critical for learning, especially in complex subjects like Earth Science. In these domains, students must explain phenomena rather than simply solve equations. However, grading open-ended, short-answer responses is a significant time burden for educators.
While previous AI efforts have focused on structured subjects like mathematics, science responses often vary wildly in vocabulary and syntax. Furthermore, existing automated systems frequently provide a score without an explanation, leaving both students and teachers without actionable feedback. This study addresses the challenge of generating accurate scores alongside pedagogically sound explanations in a data-impoverished K-12 environment.
Methodology
The researchers developed a "human-in-the-loop" framework utilizing GPT-4. The methodology consists of four primary components designed to steer the model toward human-like grading accuracy:
- Persona Prompting: The model is instructed to adopt the persona of a middle school teacher to align its tone and expectations.
- In-Context Learning (Few-Shot): Instead of computationally expensive fine-tuning, the model is provided with a small set of labeled student responses and corresponding scores within the prompt.
- Chain-of-Thought (CoT) Reasoning: Prompts include "reasoning chains" that explicitly state the logic behind a score based on the rubric (e.g., identifying specific required concepts versus student mentions).
- Active Learning (AL) Iteration: This core component involves identifying systematic model errors, selecting representative "failure" instances, and writing corrected CoT explanations to be fed back into the prompt.
Experimental Setup and Datasets
The study utilized data from the SPICE (Science Projects Integrating Computing and Engineering) curriculum, an NGSS-aligned middle school Earth Science program. The dataset included responses from 270 students at a Southeastern U.S. public middle school. Analysis focused on three specific questions (Q1, Q2, Q3) regarding water runoff and conservation of matter, with subscores for "Science Concepts" (factual accuracy) and "Scientific Reasoning" (application of principles).
Key Results and Discussion
The model's performance was evaluated using Macro F1 and Cohen’s Quadratic Weighted Kappa (QWK). The "CoT + AL" approach generally outperformed zero-shot and standard few-shot baselines, particularly on complex questions.
- High Alignment: 9 out of 11 subscores achieved "strong" agreement with human graders (QWK ≥ 0.8).
- Near Perfection: 4 subscores reached "almost perfect" agreement (QWK > 0.9).
- Complexity Handling: For the "Q2 Total Score," the QWK improved significantly from 0.65 in the zero-shot baseline to 0.87 in the final CoT + AL model.
Insights into Model Behavior
The study revealed that CoT reasoning is a "double-edged sword." While it significantly improved performance on complex logic-based questions, it occasionally led to "overfitting" on simpler factual questions where the model became overly granular. Additionally, the researchers found a correlation between human grading difficulty and AI difficulty; if human raters struggled to reach a consensus, the LLM typically struggled as well, suggesting that ambiguity in assessment design is a universal challenge.
Limitations and Future Work
Key limitations include the potential for overfitting when adding too many specific examples to a prompt and the physical constraints of LLM context windows. Furthermore, the model occasionally failed to provide evidence for positive predictions (why a student was correct), which is essential for constructive feedback.
Future research will focus on automating the selection of Active Learning examples and collaborating with teachers to evaluate how these AI-generated explanations impact student learning outcomes in real-time classroom settings.
Summary updated