Similarity-Based Content Scoring - A more Classroom-Suitable Alternative to Instance-Based Scoring?

Marie Bexte, Andrea Horbach, Torsten ZeschACL 2023 · DOI

Introduction

Automated content scoring (ACS) aims to grade short, factual student responses automatically. Traditionally, this is framed as a supervised learning problem using an "instance-based" approach, where models like BERT or Logistic Regression are trained on hundreds of labeled examples per question. However, in a typical classroom setting, teachers rarely have the luxury of grading hundreds of answers to train a system. This paper evaluates "similarity-based" scoring as a potentially more efficient and explainable alternative for real-world educational environments.

Methodology

The study compares three distinct approaches across varying amounts of training data:

  1. Instance-Based (Shallow): Logistic Regression (LR) using n-gram features.
  2. Instance-Based (Deep): A fine-tuned BERT model.
  3. Similarity-Based (Deep): A Sentence-BERT (SBERT) model.

Unlike instance-based models that predict a score directly from a single input, similarity-based models compare a student's response to a set of reference answers.

Overview of similarity-based scoring comparing learner answers to reference solutions.

As shown in the figure above, the SBERT model is fine-tuned on pairs of answers. Pairs with identical scores are labeled "similar" (1), while those with different scores are "dissimilar" (0). During inference, the system identifies the most similar training answer to a new student response and assigns the corresponding score.

Experimental Setup

The researchers utilized three major English content scoring datasets: ASAP-SAS (10 prompts), SRA-Beetle (47 prompts), and SRA-SEB (135 prompts). Performance was measured using learning curves, tracking how models improved as training data scaled from just 2 examples up to 1,000. They also conducted cross-prompt experiments to see how well models adapt knowledge from one question to another.

Key Results and Discussion

The empirical results challenge several common assumptions about similarity-based models.

The "Data Hunger" Myth

A primary hypothesis was that similarity-based models would require significantly less data because they model semantic relationships rather than just mapping inputs to labels. While SBERT generally outperformed BERT at very low data volumes, the gap was not as large as expected.

Learning curves for various models on the ASAP dataset showing QWK performance vs. training size.

As illustrated in the learning curves, instance-based BERT remains highly competitive. Once the training set reaches approximately 750 instances, BERT often catches up to or surpasses SBERT. Interestingly, simple Logistic Regression proved remarkably robust at extremely low data volumes on specific datasets, reminding researchers that complex deep learning is not always superior for small-scale tasks.

Cross-Prompt Performance and Stability

Surprisingly, instance-based BERT benefited more from cross-prompt pre-training than SBERT did. However, SBERT demonstrated higher stability; it showed a lower standard deviation in performance, meaning it is less sensitive to the specific subset of student answers a teacher chooses to grade manually for the training set—a vital trait for consistent classroom use.

The "Oracle" and Explainability

The study found that instance-based and similarity-based models often make different types of errors. An "Oracle" simulation—selecting the best model for each specific answer—showed a potential performance increase of 0.2 to 0.4 in QWK/F1. This suggests that ensemble methods combining both paradigms could be highly effective. Furthermore, similarity-based models offer a clear advantage in explainability, as they can justify a grade by pointing to a specific reference solution (e.g., "Your answer is 86% similar to this model solution").

Conclusion

While similarity-based scoring is not a "magic bullet" for small datasets, it offers distinct advantages for classroom use, particularly regarding stability and pedagogical feedback. Future work should focus on developing stacked classifiers that leverage the complementary strengths of both instance-based and similarity-based paradigms to improve overall scoring accuracy.

Summary updated

BibTeX
@inproceedings{bexte2023similarity,
  author = {Marie Bexte and Andrea Horbach and Torsten Zesch},
  title = {{Similarity-Based Content Scoring - A more Classroom-Suitable Alternative to Instance-Based Scoring?}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.119},
}
Figure