Similarity-Based Content Scoring - How to Make S-BERT Keep Up With BERT

Marie Bexte, Andrea Horbach, Torsten ZeschNAACL 2022

TL;DR

This paper introduces a novel similarity-based approach for automated essay scoring (AES) using fine-tuned Sentence-BERT (S-BERT) models. They demonstrate that this method can achieve performance comparable to traditional BERT-based instance-scoring models, even with limited training data, by effectively learning to compare student answers to reference answers.

What Problem Does This Paper Address?

Automated essay scoring (AES) systems typically rely on "instance-based" models, which learn to predict a score directly from the features of a single student's answer. An alternative, less explored approach is "similarity-based" scoring, where a student's answer is compared to a set of pre-existing reference answers. While instance-based methods are common, they often require substantial labeled data for each specific prompt. Similarity-based methods, on the other hand, are appealing when training data is scarce, as they leverage existing reference answers. However, previous comparisons (like Sakaguchi et al., 2015) suggested instance-based methods were superior. This paper investigates whether powerful neural similarity models, specifically fine-tuned S-BERT, can bridge this performance gap, especially in low-data scenarios.

Methods

The core of the proposed method is a fine-tuned Sentence-BERT (S-BERT) model. S-BERT is a modification of BERT designed to produce semantically meaningful sentence embeddings. The authors explore two main S-BERT configurations:

  • S-BERT-orig: Uses a pre-trained all-MiniLM-L6-v2 S-BERT model directly without further fine-tuning.
  • S-BERT-finetune: Fine-tunes the S-BERT model using answer pairs from the dataset. This fine-tuning process uses a CosineSimilarityLoss and a BinaryClassificationEvaluator. Positive instances are pairs of answers with the same human score (considered similar), and negative instances are pairs with different scores (considered dissimilar). This assigns a similarity score of 1 to similar pairs and 0 to dissimilar pairs, regardless of the magnitude of the score difference.

For prediction, the fine-tuned S-BERT model encodes both the student answer and reference answers into dense vectors. Cosine similarity is then computed between the student answer and each reference answer. Two strategies are used to derive a final score:

  • MAX: Assigns the score of the closest reference answer.
  • AVG: Assigns the score of the group of reference answers that has the highest average similarity to the student's answer.

The embeddings of reference answers can be pre-computed, making the scoring process efficient at test time.

Datasets

The primary dataset used is the ASAP-SAS dataset from the Kaggle short answer competition. This dataset comprises 10 prompts with approximately 2,000 answers per prompt, covering topics like Sciences, English Language Arts (ELA), and Biology. The prompts have varying scoring scales (0-2 or 0-3 points). The dataset provides scoring rubrics but no explicit set of reference answers for individual scores. The authors define "reference answers" as answers drawn from the training data pool.

They evaluate two data split scenarios:

  • Limited Data: Uses only 60 learner answers per prompt for training, sampled to ensure all score levels are represented. For similarity-based approaches, 48 answers are used for training and 12 for validation.
  • Full Data: Uses the entire training set (approximately 90% of the data). For similarity-based approaches, 100 answers are set aside for validation, and the rest are used for training.

Key Insights and Ideas

  • Fine-tuning is Crucial: The performance difference between using a pre-trained S-BERT (S-BERT-orig) and a fine-tuned S-BERT (S-BERT-finetune) is substantial, highlighting the importance of adapting the model to the specific scoring task.
  • Similarity vs. Instance-Based: The paper demonstrates that well-tuned neural similarity models can indeed compete with traditional instance-based methods, challenging prior assumptions.
  • AVG Strategy Gains Traction: While the MAX strategy initially performed better with the pre-trained model, the AVG strategy becomes more competitive, and even slightly superior, with the fine-tuned S-BERT, suggesting a more robust aggregation of similarity information.
  • Prompt-Specific Learning is Key: Cross-prompt evaluation reveals that similarity functions learned on one prompt do not generalize well to others, even within the same topic. This indicates that prompt-specific fine-tuning is essential for achieving good performance.
  • Limited Data Viability: The similarity-based approach shows promise even with very limited training data (60 answers), outperforming a BERT-based instance-scoring model in the limited data setting.

Key Results

  • Overall Performance: On the ASAP-SAS dataset, the fine-tuned S-BERT approach (both MAX and AVG) achieves Quadratic Weighted Kappa (QWK) scores comparable to, and in some cases exceeding, a BERT-based instance-scoring model, especially in the "Full Data" setting. For example, S-BERT-finetune AVG achieves a QWK of 0.72 with full data, on par with BERT (0.74).
  • Limited Data Performance: In the "Limited Data" setting, the fine-tuned S-BERT (AVG: 0.43, MAX: 0.44) outperforms the BERT instance-based model (0.37), though a Logistic Regression baseline (0.50) performs best in this specific low-data scenario.
  • Cross-Prompt Performance: Cross-prompt evaluation shows significantly lower QWK scores, indicating that models are highly prompt-dependent and do not generalize well across different prompts, even within the same topic. For instance, Biology cross-prompt QWK is around 0.18-0.40, much lower than direct fine-tuning (0.70+).

Limitations and Future Work

  • Prompt Dependency: The study acknowledges that similarity models are highly prompt-dependent, and generalization across prompts is poor. This suggests that effective training data selection for each prompt is crucial.
  • Cross-Prompt Hypothesis: The initial hypothesis that models trained on the same topic would generalize better was not strongly supported. The results suggest that fine-tuning on a different prompt, even from the same topic, does not necessarily yield better results than fine-tuning on a completely unrelated prompt.
  • Future Work: The authors suggest further research into comparing the two paradigms (instance-based vs. similarity-based) more extensively, particularly concerning varying amounts of training data and exploring different datasets to better understand which paradigm is preferable under various conditions. They also plan to investigate this architecture further for challenges like the NAEP-AS challenge.

Summary updated

BibTeX
@inproceedings{bexte2022similarity,
  author = {Marie Bexte and Andrea Horbach and Torsten Zesch},
  title = {{Similarity-Based Content Scoring - How to Make S-BERT Keep Up With BERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure