Automatic Text Scoring Using Neural Networks

Dimitrios Alikaniotis, Helen Yannakoudakis, Marek Rei

Introduction

Automated Text Scoring (ATS) is a critical component of modern large-scale educational assessment. By providing a cost-effective and consistent alternative to human grading, ATS systems enable rapid feedback for students and reduced overhead for institutions. However, traditional ATS systems have historically relied on "feature-heavy" models. These require linguists and engineers to manually design features that capture complex linguistic properties like grammar, vocabulary richness, and discourse coherence.

This manual feature engineering is labor-intensive and often fails to generalize across different essay prompts or genres. This paper presents a deep learning framework that eliminates the need for hand-crafted features by learning them directly from raw text. The proposed approach combines a novel "Score-Specific Word Embedding" (SSWE) technique with Long Short-Term Memory (LSTM) networks to achieve state-of-the-art performance.

Methods: A Two-Stage Neural Architecture

The researchers developed a two-stage neural architecture designed to capture both the semantic meaning of words and the sequential flow of an essay.

1. Score-Specific Word Embeddings (SSWE)

Standard word embeddings, such as Word2Vec, represent words in a high-dimensional vector space based on their linguistic context. While effective for many NLP tasks, these embeddings can be counterproductive for grading. For instance, a misspelling like "copmuter" often appears in the same context as "computer," leading standard models to treat them as nearly identical.

To address this, the authors introduced Score-Specific Word Embeddings (SSWE). This technique augments the embedding training process with a regression unit that forces the model to learn the quality or "score" associated with a word. As a result, the model learns to separate high-quality vocabulary from common errors or low-quality usage in the vector space.

Comparison of standard neural embeddings (a) and score-specific word embeddings (b). Note how SSWE separates misspellings from correct words based on their impact on the final score.

2. Bi-directional LSTM Networks

The SSWEs serve as input for a Long Short-Term Memory (LSTM) network. LSTMs are a type of recurrent neural network specifically designed to process sequences, making them ideal for capturing the compositionality and "flow" of an essay.

The system utilizes a Bi-directional LSTM (BLSTM), which processes the text in both forward and backward directions. This allows the model to incorporate both "past" and "future" context for every word in a sentence, leading to a more robust representation of the essay's overall structure. The final representation is then passed through a linear layer to predict the essay's score.

A Bi-directional LSTM architecture processing a word sequence. The model reads the text in both directions to capture comprehensive context.

Evaluation and Results

The model was evaluated using the Kaggle Automated Student Assessment Prize (ASAP) dataset, which contains approximately 12,976 essays across eight distinct prompts. The performance was measured using Quadratic Weighted Kappa ($\kappa$), a standard metric for inter-rater agreement.

The SSWE + Two-layer BLSTM model significantly outperformed traditional baselines:

  • Agreement: The model achieved a Quadratic Weighted Kappa of 0.96, indicating that its grading is nearly indistinguishable from human raters.
  • Correlation: It reached a Spearman’s $\rho$ of 0.91, far exceeding the baseline Support Vector Machine (SVM) score of 0.78.
  • Error Reduction: The Root Mean Square Error (RMSE) was reduced to 2.4, compared to 8.85 for the feature-based SVM.

Insights and Future Directions

A key insight from this research is that linguistic context is not the only factor in text quality; usage information (like correct spelling) is a powerful signal that standard embeddings often ignore. By integrating score information directly into the word representations, the model gains a significant advantage.

Furthermore, the authors introduced a gradient-based visualization method to help interpret the model's decisions. By highlighting words that contribute most to a high or low score, the system provides a degree of transparency often missing in deep learning models.

However, challenges remain. The model occasionally struggles with rare words or proper names, sometimes flagging them as low quality. Additionally, while the system can identify which parts of an essay are problematic, it does not yet provide granular feedback on why (e.g., distinguishing between a grammatical error and a logical flaw). Future work will focus on improving the granularity of feedback to make these systems even more valuable for student learning.

Summary updated

Abstract

Automated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking. However, in order to achieve good performance, the predictive features of the system need to be manually engineered by human experts. We introduce a model that forms word representations by learning the extent to which specific words contribute to the text's score. Using Long-Short Term Memory networks to represent the meaning of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches. In an attempt to make our results more interpretable, and inspired by recent advances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative.
BibTeX
@inproceedings{alikaniotis2016automatic,
  author = {Dimitrios Alikaniotis and Helen Yannakoudakis and Marek Rei},
  title = {{Automatic Text Scoring Using Neural Networks}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2016},
  url = {https://openreview.net/forum?id=Sk-lG2gdZB},
}
Figure