TL;DR
This paper tackles the challenge of automatically assessing the quality of student writing revisions. They introduce a new annotated corpus of student revisions and demonstrate a machine learning model that can predict whether a revision improves an essay. Crucially, they show that combining student revisions with expert edits significantly boosts performance, especially for identifying less-than-ideal revisions.
What Problem Does This Paper Address?
Improving student writing is a key goal in education, and revision is a critical part of that process. While writing assistant tools exist, they often fail to explicitly tell students if their revisions are actually making their writing better. Evaluating revision quality is time-consuming for human instructors. This paper addresses the gap of automatically assessing revision quality at the sentence level, aiming to build systems that can provide students with feedback on whether their edits are genuinely improving their argumentative essays.
Methods
The authors employ a machine learning approach to classify sentence-level revisions as either "Better" or "NotBetter." They start by adapting features commonly used in prior NLP revision analysis tasks, such as unigram, bigram, and trigram counts, sentence length differences, and edit distance. They then introduce new features tailored to revision quality, including specificity measures (using the Speciteller tool) and parse tree features to capture readability and coherence. To train their models, they utilize a combination of student-generated revisions and expert-edited revisions, exploring different sampling strategies. They use a Random Forest classifier and evaluate performance using 10-fold cross-validation, reporting precision, recall, and F1 scores.
Datasets
- ArgRewrite Corpus: A subset of revisions from student argumentative essays. This is the primary source for their newly annotated "non-expert" revisions.
- Automated Evaluation of Scientific Writing (AESW) Corpus: This dataset contains expert-edited scientific writing. The authors leverage this to extract "expert" revisions to augment their training data. They use two samples: "AESW all" (all genres) and "AESW plaintext" (ignoring sentences with placeholders).
Key Insights and Ideas
- Revision Quality is Hard: Even with clear guidelines, annotating revision quality is challenging, as evidenced by the "slight agreement" initially observed among annotators.
- Expert Edits are Valuable: Combining expert-edited revisions with student revisions significantly improves the model's ability to predict revision quality, particularly for identifying "NotBetter" revisions. This suggests that expert edits provide a valuable signal for what constitutes improvement.
- Length is a Deceptive Feature: While length difference is a strong predictor, it can be misleading, potentially causing models to favor longer sentences as "Better" even when they aren't. This highlights the need for more nuanced features.
- Context Matters: The misclassification of some "NotBetter" revisions suggests that sentence-level analysis might be insufficient, and incorporating paragraph-level context could improve performance.
Key Results
- A model trained solely on ArgRewrite data outperformed a majority baseline, achieving higher precision and F1 scores. However, its recall for "NotBetter" revisions was low.
- A model trained solely on AESW plaintext data showed significantly higher recall for "NotBetter" revisions compared to other models.
- The best performing model combined ArgRewrite data with AESW plaintext data, achieving the highest overall classification performance across precision, recall, and F1.
- Key features contributing to prediction include unigrams, trigrams, length difference, language errors, edit distance, BLEU score, specificity difference, and parse-tree features.
Limitations and Future Work
The authors acknowledge that their current model's performance in identifying "NotBetter" revisions is still not ideal. They also note that the reliance on sentence-level analysis might be a limitation, as context could be crucial.
Future work includes:
- Improving inter-rater reliability by collecting annotations from experts rather than crowdsourcing.
- Investigating how feature extraction algorithms impact feature utility.
- Exploring ways to improve the use of AESW data, such as clustering revisions for more targeted sampling.
- Optimizing the quantity of AESW revisions used and how to balance labels during sampling.
- Potentially incorporating revision context to improve classification.
Summary updated