Abstract
Automated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking. However, in order to achieve good performance, the predictive features of the system need to be manually engineered by human experts. We introduce a model that forms word representations by learning the extent to which specific words contribute to the text's score. Using Long-Short Term Memory networks to represent the meaning of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches. In an attempt to make our results more interpretable, and inspired by recent advances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative.
Introduction
Automated Text Scoring (ATS) is the task of using algorithms to grade essays or open-ended responses. Historically, this required "feature engineering"—human experts manually defining rules for grammar, vocabulary diversity, and sentence structure. This process is labor-intensive and often fails to generalize across different essay prompts or domains.
While neural networks can learn these features automatically, they are often criticized for being "black boxes." In high-stakes educational settings, it isn't enough to just give a score; we need to understand why a model gave that score. This paper tackles both the need for automated feature learning and the need for model interpretability.
Methodology
The authors propose a two-stage neural architecture that combines novel word representations with recurrent neural networks.
Score-Specific Word Embeddings (SSWE)
Standard word embeddings (like Word2Vec) group words based on local context. For instance, "apple" and "orange" appear in similar sentences and thus end up close in vector space. However, in ATS, a misspelling like "copmuter" might share the same context as "computer" but indicates a very different quality level of writing.
The authors augmented the classic Collobert & Weston (C&W) model to learn embeddings that account for the score of the text. The loss function is a weighted combination of predicting the surrounding words (context) and predicting the overall essay score. This allows the model to differentiate between words that are linguistically similar but pedagogically distinct.
Bi-directional LSTM Networks
These embeddings are fed into a multi-layer Bi-directional LSTM (Bi-LSTM). The Bi-LSTM processes the essay from both beginning-to-end and end-to-beginning, capturing long-distance dependencies and global context.
The final hidden states from both directions are concatenated and passed through a linear regression layer to predict the final score.
Visualization via Gradients
To interpret the model, the authors used a "pseudo-score" method. They calculated the gradient of the loss function with respect to the input word vectors. By seeing how much a word would need to change to reach a "perfect" score versus a "failing" score, they could highlight which words the model viewed as high-quality or problematic.
Experimental Setup
The study used the Kaggle ASAP (Automated Student Assessment Prize) dataset, consisting of 12,976 essays written by students in Grades 7 through 10. The dataset contains eight distinct sets based on different prompts, each with its own marking criteria and score range. Ground truth scores were established by double-marking by human raters, with expert resolution for discrepancies.
Results and Discussion
The model significantly outperformed traditional baselines like Support Vector Regression (SVR) and standard LSTMs.
| Model | Spearman’s $\rho$ | Pearson $r$ | RMSE |
|---|---|---|---|
| SVM (Manual Features) | 0.78 | 0.77 | 8.85 |
| Standard LSTM | 0.59 | 0.60 | 6.80 |
| SSWE + Two-layer BLSTM | 0.91 | 0.96 | 2.40 |
The SSWE + Two-layer BLSTM achieved a Pearson correlation of 0.96, suggesting the model's predictions are nearly indistinguishable from human gold standards.
Key Insights
- The "Copmuter" Effect: As shown in the embedding visualization, SSWEs "pull" misspellings apart from correct versions in vector space because the misspelling is a strong signal for a lower score.
- Context vs. Usage: By tuning a hyperparameter ($\alpha$), the authors balanced linguistic context versus score-specific usage. A heavy lean toward score-specific information ($\alpha = 0.1$) yielded the best results.
- Interpretability through "Stress": The visualization method identifies words that "stress" the prediction of a high score, providing a window into the model's internal logic.
Conclusion
This research demonstrates that a deep learning framework can achieve state-of-the-art performance in ATS without manual feature engineering. While the model currently struggles with rare proper nouns and lacks granular pedagogical feedback, the introduction of SSWEs and gradient-based visualization marks a significant step toward more accurate and interpretable automated grading systems.
Summary updated