A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science

Clayton Cohn, Nicole Hutchins, Tuan Le, Gautam BiswasAAAI 2024 · DOI

Abstract

Automating the evaluation of open-ended science assessments remains a challenge due to the non-canonical nature of student responses. This paper introduces a human-in-the-loop framework that leverages GPT-4, Chain-of-Thought (CoT) prompting, and Active Learning (AL) to score and explain middle school science responses. Our results demonstrate that this approach achieves high alignment with human graders (QWK ≥ 0.8) and provides pedagogically sound reasoning, offering a scalable solution for formative feedback in K-12 STEM education.

Introduction

Formative assessment—the frequent, low-stakes checking of student understanding—is critical for learning, especially in complex subjects like Earth Science. In these domains, students must explain phenomena rather than simply solve equations. However, grading open-ended, short-answer responses is a significant time burden for educators.

While previous AI efforts have focused on structured subjects like mathematics, science responses often vary wildly in vocabulary and syntax. Furthermore, existing automated systems frequently provide a score without an explanation, leaving both students and teachers without actionable feedback. This study addresses the challenge of generating accurate scores alongside pedagogically sound explanations in a data-impoverished K-12 environment.

Methodology

The researchers developed a "human-in-the-loop" framework utilizing GPT-4. The methodology consists of four primary components designed to steer the model toward human-like grading accuracy:

  1. Persona Prompting: The model is instructed to adopt the persona of a middle school teacher to align its tone and expectations.
  2. In-Context Learning (Few-Shot): Instead of computationally expensive fine-tuning, the model is provided with a small set of labeled student responses and corresponding scores within the prompt.
  3. Chain-of-Thought (CoT) Reasoning: Prompts include "reasoning chains" that explicitly state the logic behind a score based on the rubric (e.g., identifying specific required concepts versus student mentions).
  4. Active Learning (AL) Iteration: This core component involves identifying systematic model errors, selecting representative "failure" instances, and writing corrected CoT explanations to be fed back into the prompt.
The human-in-the-loop framework combining CoT prompting and Active Learning.

Experimental Setup and Datasets

The study utilized data from the SPICE (Science Projects Integrating Computing and Engineering) curriculum, an NGSS-aligned middle school Earth Science program. The dataset included responses from 270 students at a Southeastern U.S. public middle school. Analysis focused on three specific questions (Q1, Q2, Q3) regarding water runoff and conservation of matter, with subscores for "Science Concepts" (factual accuracy) and "Scientific Reasoning" (application of principles).

Key Results and Discussion

The model's performance was evaluated using Macro F1 and Cohen’s Quadratic Weighted Kappa (QWK). The "CoT + AL" approach generally outperformed zero-shot and standard few-shot baselines, particularly on complex questions.

  • High Alignment: 9 out of 11 subscores achieved "strong" agreement with human graders (QWK ≥ 0.8).
  • Near Perfection: 4 subscores reached "almost perfect" agreement (QWK > 0.9).
  • Complexity Handling: For the "Q2 Total Score," the QWK improved significantly from 0.65 in the zero-shot baseline to 0.87 in the final CoT + AL model.
Performance metrics across different prompting strategies for Question 2 subscores, showing the benefit of CoT and Active Learning.

Insights into Model Behavior

The study revealed that CoT reasoning is a "double-edged sword." While it significantly improved performance on complex logic-based questions, it occasionally led to "overfitting" on simpler factual questions where the model became overly granular. Additionally, the researchers found a correlation between human grading difficulty and AI difficulty; if human raters struggled to reach a consensus, the LLM typically struggled as well, suggesting that ambiguity in assessment design is a universal challenge.

Limitations and Future Work

Key limitations include the potential for overfitting when adding too many specific examples to a prompt and the physical constraints of LLM context windows. Furthermore, the model occasionally failed to provide evidence for positive predictions (why a student was correct), which is essential for constructive feedback.

Future research will focus on automating the selection of Active Learning examples and collaborating with teachers to evaluate how these AI-generated explanations impact student learning outcomes in real-time classroom settings.

Summary updated

BibTeX
@inproceedings{cohn2024chain,
  author = {Clayton Cohn and Nicole Hutchins and Tuan Le and Gautam Biswas},
  title = {{A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30364},
}
Figure