Abstract
Deep learning has demonstrated tremendous potential for Automatic Text Scoring (ATS) tasks. In this paper, we describe a new neural architecture that enhances vanilla neural network models with auxiliary neural coherence features. Our new method proposes a new SkipFlow mechanism that models relationships between snapshots of the hidden representations of a long short-term memory (LSTM) network as it reads. Subsequently, the semantic relationships between multiple snapshots are used as auxiliary features for prediction. This has two main benefits. Firstly, essays are typically long sequences and therefore the memorization capability of the LSTM network may be insufficient. Implicit access to multiple snapshots can alleviate this problem by acting as a protection against vanishing gradients. The parameters of the SkipFlow mechanism also acts as an auxiliary memory. Secondly, modeling relationships between multiple positions allows our model to learn features that represent and approximate textual coherence. In our model, we call this neural coherence features. Overall, we present a unified deep learning architecture that generates neural coherence features as it reads in an end-to-end fashion. Our approach demonstrates state-of-the-art performance on the benchmark ASAP dataset, outperforming not only feature engineering baselines but also other deep learning models.
Introduction
Automatic Text Scoring (ATS) is a high-stakes task used in exams like the TOEFL and GRE. Historically, this relied on "handcrafted" features—manually designed metrics for grammar, length, and coherence. While deep learning models like LSTMs (Long Short-Term Memory) and CNNs (Convolutional Neural Networks) removed the need for manual engineering, they introduced new challenges. Standard LSTMs struggle with the sheer length of essays (often 300–800 words), leading to "forgetting" or vanishing gradients. More importantly, these models typically compress an entire essay into a single vector, failing to explicitly capture textual coherence—the logical progression and semantic connectivity between different parts of a text. SkipFlow aims to bridge this gap by allowing the model to "look back" and compare different stages of the reading process.
Methodology
The SkipFlow LSTM augments a standard recurrent neural network with a specialized mechanism for generating "neural coherence features."
- The Base Model: The system starts with a standard embedding layer and an LSTM that reads the essay word-by-word.
- The SkipFlow Mechanism: Instead of only using the final output, the model takes "snapshots" of the LSTM’s hidden state at a fixed interval, called the relevance width ($\delta$).
- Neural Tensor Layer (NTL): This is the core innovation. The NTL takes two snapshots ($h_i$ and $h_{i+\delta}$) and computes a similarity score. Unlike a simple dot product, the NTL uses a 3D tensor to model multiple instances of second-order interactions between the two vectors. This allows the model to learn complex, non-linear relationships that represent how the "meaning" of the essay is evolving.
- Aggregation: These similarity scores (the neural coherence features) are concatenated into a vector and combined with the mean-pooled representation of the entire LSTM. This combined "super-vector" is then passed through a final hidden layer to predict the essay score.
Experimental Setup
The model was evaluated on the ASAP (Automated Student Assessment Prize) dataset, a standard benchmark in the field. The dataset includes 8 different essay prompts ranging from argumentative to narrative genres, with approximately 700 to 1,800 essays per prompt. Performance was measured using the Quadratic Weighted Kappa (QWK), which calculates the agreement between the automated score and human graders.
Results and Discussion
SkipFlow (Tensor) achieved an average QWK of 0.764, outperforming the best non-deep learning baseline (EASE) by about 6% and beating massive ensembles of 20 models from previous state-of-the-art research. Despite the added complexity of the tensor layer, SkipFlow is computationally efficient, adding only 1–2 seconds per training epoch compared to a vanilla LSTM.
Key Insights
- Coherence as a Latent Feature: Coherence is captured through the relationship between latent semantic representations learned end-to-end.
- Relieving the Recurrent Burden: Auxiliary features from intermediate states act as a safeguard against the vanishing gradient problem in long sequences.
- Hyperparameter Sensitivity: A "relevance width" ($\delta$) of 50 to 100 worked best. If the width is too small, the model is overwhelmed by local comparisons; if too large, it misses the flow of the argument.
Limitations and Future Work
The current model uses a fixed relevance width, though logical transitions in writing don't always happen at fixed intervals. Future iterations might benefit from a dynamic or attention-based skipping mechanism. Additionally, while the neural coherence features improve scores, it remains difficult to interpret what the tensor layer is actually seeing. Future work could focus on visualizing these features to provide feedback to students on their essay flow.
Summary updated