Introduction
Automated content scoring (ACS) aims to grade short, factual student responses automatically. Traditionally, this is framed as a supervised learning problem using an "instance-based" approach, where models like BERT or Logistic Regression are trained on hundreds of labeled examples per question. However, in a typical classroom setting, teachers rarely have the luxury of grading hundreds of answers to train a system. This paper evaluates "similarity-based" scoring as a potentially more efficient and explainable alternative for real-world educational environments.
Methodology
The study compares three distinct approaches across varying amounts of training data:
- Instance-Based (Shallow): Logistic Regression (LR) using n-gram features.
- Instance-Based (Deep): A fine-tuned BERT model.
- Similarity-Based (Deep): A Sentence-BERT (SBERT) model.
Unlike instance-based models that predict a score directly from a single input, similarity-based models compare a student's response to a set of reference answers.
As shown in the figure above, the SBERT model is fine-tuned on pairs of answers. Pairs with identical scores are labeled "similar" (1), while those with different scores are "dissimilar" (0). During inference, the system identifies the most similar training answer to a new student response and assigns the corresponding score.
Experimental Setup
The researchers utilized three major English content scoring datasets: ASAP-SAS (10 prompts), SRA-Beetle (47 prompts), and SRA-SEB (135 prompts). Performance was measured using learning curves, tracking how models improved as training data scaled from just 2 examples up to 1,000. They also conducted cross-prompt experiments to see how well models adapt knowledge from one question to another.
Key Results and Discussion
The empirical results challenge several common assumptions about similarity-based models.
The "Data Hunger" Myth
A primary hypothesis was that similarity-based models would require significantly less data because they model semantic relationships rather than just mapping inputs to labels. While SBERT generally outperformed BERT at very low data volumes, the gap was not as large as expected.
As illustrated in the learning curves, instance-based BERT remains highly competitive. Once the training set reaches approximately 750 instances, BERT often catches up to or surpasses SBERT. Interestingly, simple Logistic Regression proved remarkably robust at extremely low data volumes on specific datasets, reminding researchers that complex deep learning is not always superior for small-scale tasks.
Cross-Prompt Performance and Stability
Surprisingly, instance-based BERT benefited more from cross-prompt pre-training than SBERT did. However, SBERT demonstrated higher stability; it showed a lower standard deviation in performance, meaning it is less sensitive to the specific subset of student answers a teacher chooses to grade manually for the training set—a vital trait for consistent classroom use.
The "Oracle" and Explainability
The study found that instance-based and similarity-based models often make different types of errors. An "Oracle" simulation—selecting the best model for each specific answer—showed a potential performance increase of 0.2 to 0.4 in QWK/F1. This suggests that ensemble methods combining both paradigms could be highly effective. Furthermore, similarity-based models offer a clear advantage in explainability, as they can justify a grade by pointing to a specific reference solution (e.g., "Your answer is 86% similar to this model solution").
Conclusion
While similarity-based scoring is not a "magic bullet" for small datasets, it offers distinct advantages for classroom use, particularly regarding stability and pedagogical feedback. Future work should focus on developing stacked classifiers that leverage the complementary strengths of both instance-based and similarity-based paradigms to improve overall scoring accuracy.
Summary updated