Abstract
Automated Essay Scoring (AES) has traditionally relied on extensive manual feature engineering. This paper, presented at EMNLP 2016, introduces the first end-to-end neural network architecture for AES that requires zero manual feature engineering. By utilizing a Long Short-Term Memory (LSTM) model with a "mean-over-time" (MoT) aggregation layer, the authors achieve a significant performance boost over traditional feature-based baselines, reaching levels of agreement comparable to human graders.
Introduction
Traditional AES systems, such as those developed by ETS, have historically depended on "hand-crafted" features like essay length, grammar counts, and part-of-speech distributions. This approach is not only laborious, requiring significant domain expertise, but also limited by the features selected; if a human designer overlooks a critical aspect like "argumentative flow," the system cannot account for it. This research addresses these flaws by treating essays as raw sequences of data and allowing the model to automatically learn the relationship between text and score.
Methods: Neural Architecture
The core contribution is a Convolutional Recurrent Neural Network designed specifically for long-form text. The architecture processes an essay through several specialized layers:
- Lookup Table: Maps words to dense 50-dimensional embeddings.
- Convolution Layer: Optionally extracts local n-gram features via a sliding window.
- Recurrent Layer (LSTM): Processes the sequence while maintaining a "memory" of the text, which is vital for long essays.
- Mean-over-Time (MoT) Layer: Instead of relying solely on the final hidden state—which may "forget" early parts of a long essay—this layer averages all intermediate hidden states to ensure the entire content is considered equally.
- Output Layer: A linear layer with Sigmoid activation maps the representation to a continuous score between 0 and 1.
Experimental Setup
The models were evaluated using the ASAP (Automated Student Assessment Prize) dataset, which contains approximately 13,000 essays across 8 different prompts. These prompts cover various genres (persuasive, narrative) and have diverse average lengths (150 to 650 words) and score ranges (0–3 to 0–60). Training utilized RMSProp to minimize Mean Squared Error, with Dropout applied to prevent overfitting on the relatively small dataset.
Results and Analysis
The neural approach demonstrated clear superiority over traditional methods. The best-performing ensemble model achieved an average Quadratic Weighted Kappa (QWK) of 0.761, representing a 5.6% improvement over the strong Bayesian Linear Ridge Regression (BLRR) baseline.
Key Insights
- The MoT Advantage: The MoT layer proved essential. Using only the final LSTM state resulted in a poor QWK of 0.540, whereas adding MoT increased it to 0.746. This confirms that grading requires a global view of the text rather than just the conclusion.
- LSTM vs. RNN: Basic RNNs were found to be ineffective due to the vanishing gradient problem, making them unable to capture dependencies in long essays compared to LSTMs and GRUs.
- Scoring Dynamics: Analysis of the model's internal state revealed that it initially assigns low scores to short essays. As length increases, the model begins to discriminate based on content, driving the score based on the quality of the writing.
- Ensembling: Combining CNNs (for local patterns like word choice) and LSTMs (for global structure) yielded the highest overall accuracy.
Conclusion
This work marks a milestone in AES by demonstrating that deep learning can outperform hand-crafted feature engineering. While the model achieves human-level agreement, challenges remain regarding data scarcity and interpretability. Future work may involve more complex attention mechanisms and hierarchical modeling to better capture essay organization and provide more actionable feedback to students.
Citation: Kaveh Taghipour, Hwee Tou Ng. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. 2016.
Summary updated