Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring

Jiaqi Lun, Jia Zhu, Yong Tang, Min YangAAAI 2020 · OpenReview

Abstract

Automatic short answer scoring (ASAS) is a research subject of intelligent education, which is a hot field of natural language understanding. Many experiments have confirmed that the ASAS system is not good enough, because its performance is limited by the training data. Focusing on the problem, we propose MDA-ASAS, multiple data augmentation strategies for improving performance on automatic short answer scoring. MDA-ASAS is designed to learn language representation enhanced by data augmentation strategies, which includes back-translation, correct answer as reference answer, and swap content. We argue that external knowledge has a profound impact on the ASAS process. Meanwhile, the Bidirectional Encoder Representations from Transformers (BERT) model has been shown to be effective for improving many natural language processing tasks, which acquires more semantic, grammatical and other features in large amounts of unsupervised data, and actually adds external knowledge. Combining with the latest BERT model, our experimental results on the ASAS dataset show that MDA-ASAS brings a significant gain over state-of-art. We also perform extensive ablation studies and suggest parameters for practical use.

Introduction

Automatic Short Answer Scoring (ASAS) is a critical component of intelligent tutoring systems, requiring AI to understand the semantic relationship between a student’s response and a teacher’s reference answer. The primary hurdle in ASAS is data scarcity; high-quality, human-graded datasets are small and expensive to produce. Furthermore, student answers are often very short, providing little contextual information for traditional NLP models. To address this, we propose MDA-ASAS, which integrates three specific augmentation strategies with a fine-tuned BERT-base model to enhance performance and robustness.

Methodology

The MDA-ASAS framework leverages the pre-trained semantic knowledge of BERT and enhances it through three data augmentation techniques: back-translation, using correct answers as reference answers, and content swapping.

Back-translation

To create paraphrased versions of the training data, English sentences are translated into Japanese and then back into English using the Baidu Translation API. This generates "twin" data that retains the original meaning but uses different syntax and vocabulary, helping the model become robust to phrasing variations.

The back-translation process used to generate paraphrased training data.

Correct Answers as Reference Answers

This novel heuristic hypothesizes that if a student’s answer receives a perfect score, it is functionally equivalent to the teacher's reference answer. By treating these high-scoring responses as new "gold standard" references, the system exponentially increases the number of valid training pairs.

Data augmentation strategy using correct student answers as new reference keys.

Swap Content

To further diversify the dataset, the authors randomly swapped columns (questions, reference answers, or student answers) between the original dataset and the "twin" dataset generated via back-translation. This creates eight different combinations of original and paraphrased text for every entry.

Experimental Results

The study utilized the SemEval-2013 dataset (SciEntsBank corpus), focusing on the Unseen Answers (UA) test set. MDA-ASAS achieved a new state-of-the-art on this dataset:

  • 2-way Classification: Reached an accuracy of 82.77%, a significant jump over the previous best of 79.26%.
  • 3-way Classification: Achieved 76.31% accuracy.
  • 5-way Classification: Achieved 70.02% accuracy.

Ablation studies showed that the "Swap Content" strategy (specifically the B4-swap-A-B1 configuration) was the most effective individual augmentation method, gaining roughly 2 points in macro-averaged F1 score over the baseline. Furthermore, the optimal context window for BERT in this task was found to be 90 words.

Conclusion

MDA-ASAS demonstrates that data augmentation is a powerful tool for overcoming data scarcity in specialized NLP tasks like ASAS. By combining sentence-level paraphrasing with logical data recycling, the framework significantly improves the accuracy of automatic grading. Future work will extend these tests to unseen questions and domains to further evaluate the model's generalizability.

Summary updated

Abstract

Automatic short answer scoring (ASAS) is a research subject of intelligent education, which is a hot field of natural language understanding. Many experiments have confirmed that the ASAS system is not good enough, because its performance is limited by the training data. Focusing on the problem, we propose MDA-ASAS, multiple data augmentation strategies for improving performance on automatic short answer scoring. MDA-ASAS is designed to learn language representation enhanced by data augmentation strategies, which includes back-translation, correct answer as reference answer, and swap content. We argue that external knowledge has a profound impact on the ASAS process. Meanwhile, the Bidirectional Encoder Representations from Transformers (BERT) model has been shown to be effective for improving many natural language processing tasks, which acquires more semantic, grammatical and other features in large amounts of unsupervised data, and actually adds external knowledge. Combining with the latest BERT model, our experimental results on the ASAS dataset show that MDA-ASAS brings a significant gain over state-of-art. We also perform extensive ablation studies and suggest parameters for practical use.
BibTeX
@inproceedings{lun2020multiple,
  author = {Jiaqi Lun and Jia Zhu and Yong Tang and Min Yang},
  title = {{Multiple Data Augmentation Strategies for Improving Performance on Automatic Short Answer Scoring}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=6DuJuke8meA},
}
Figure