SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring

Yi Tay, Minh C. Phan, Luu Anh Tuan, Siu Cheung Hui

Abstract

Deep learning has demonstrated tremendous potential for Automatic Text Scoring (ATS) tasks. In this paper, we describe a new neural architecture that enhances vanilla neural network models with auxiliary neural coherence features. Our new method proposes a new SkipFlow mechanism that models relationships between snapshots of the hidden representations of a long short-term memory (LSTM) network as it reads. Subsequently, the semantic relationships between multiple snapshots are used as auxiliary features for prediction. This has two main benefits. Firstly, essays are typically long sequences and therefore the memorization capability of the LSTM network may be insufficient. Implicit access to multiple snapshots can alleviate this problem by acting as a protection against vanishing gradients. The parameters of the SkipFlow mechanism also acts as an auxiliary memory. Secondly, modeling relationships between multiple positions allows our model to learn features that represent and approximate textual coherence. In our model, we call this neural coherence features. Overall, we present a unified deep learning architecture that generates neural coherence features as it reads in an end-to-end fashion. Our approach demonstrates state-of-the-art performance on the benchmark ASAP dataset, outperforming not only feature engineering baselines but also other deep learning models.

Introduction

Automated Text Scoring (ATS) is a critical task in educational technology, aimed at providing scalable, objective, and immediate feedback on student writing. While traditional methods relied heavily on handcrafted features—manually designed rules for grammar, syntax, and discourse—modern deep learning approaches like LSTMs and CNNs have shifted the focus toward automated feature extraction.

However, standard deep learning models face two significant hurdles in essay scoring:

  1. Vanishing Coherence: LSTMs excel at local word-to-word dependencies but often fail to capture the global "flow" or logical transitions across an entire essay, which can span hundreds of words.
  2. Memory Bottlenecks: In long sequences, the influence of early text segments tends to "vanish" by the time the model reaches the end, making it difficult to maintain a holistic view of the essay's quality.

The SkipFlow Architecture

To address these issues, we introduce the SkipFlow LSTM. This architecture augments a standard recurrent network with a specialized mechanism designed to track semantic relationships across time, effectively modeling "neural coherence."

The SkipFlow architecture incorporates an LSTM layer with a SkipFlow mechanism that uses a Neural Tensor Layer to extract coherence features from hidden state snapshots.

SkipFlow Mechanism and Neural Tensor Layer

The core innovation is the SkipFlow mechanism. As the LSTM reads the essay, the model takes "snapshots" of its hidden states at fixed intervals, defined by a hyperparameter $\delta$ (the "relevance width").

For every pair of these snapshots, a Neural Tensor Layer (NTL) calculates a similarity score. Unlike simple dot products, the NTL utilizes a 3D tensor to model complex, multi-faceted interactions between the hidden representations. These similarity scores are then concatenated into a "neural coherence feature vector." This vector provides the final scoring layer with direct access to the logical flow of the text, serving as an auxiliary memory that mitigates the vanishing gradient problem.

Experimental Evaluation

We evaluated SkipFlow on the ASAP (Automated Student Assessment Prize) dataset, which contains eight essay prompts across various genres (argumentative, narrative, etc.). Performance was measured using the Quadratic Weighted Kappa (QWK), a standard metric for agreement between AI and human graders.

Results and Hyperparameter Sensitivity

The SkipFlow LSTM (Tensor) achieved an average QWK of 0.764, a significant improvement over vanilla LSTM models (0.646 QWK) and traditional feature-engineering baselines like EASE.

Impact of the relevance width $\delta$ on the QWK score across different essay prompts (P4-P7).

A key finding was the impact of the relevance width $\delta$. As shown in the experimental results, performance generally improves or stabilizes as $\delta$ increases, suggesting that comparing more distant snapshots helps the model capture broader coherence patterns. Interestingly, the SkipFlow mechanism provided substantial gains for LSTMs but had minimal impact on CNNs, indicating its specific synergy with temporal recurrent modeling.

Conclusion

SkipFlow demonstrates that explicitly modeling neural coherence through temporal snapshots can significantly enhance the performance of automated text scoring. By treating coherence as a measurable relationship between distant points in a text, the model mimics human grading criteria more closely than standard recurrent networks. Future work will explore dynamic "skipping" mechanisms and methods to translate these coherence scores into actionable feedback for students.

Summary updated

Abstract

Deep learning has demonstrated tremendous potential for Automatic Text Scoring (ATS) tasks. In this paper, we describe a new neural architecture that enhances vanilla neural network models with auxiliary neural coherence features. Our new method proposes a new \textsc{SkipFlow} mechanism that models relationships between snapshots of the hidden representations of a long short-term memory (LSTM) network as it reads. Subsequently, the semantic relationships between multiple snapshots are used as auxiliary features for prediction. This has two main benefits. Firstly, essays are typically long sequences and therefore the memorization capability of the LSTM network may be insufficient. Implicit access to multiple snapshots can alleviate this problem by acting as a protection against vanishing gradients. The parameters of the \textsc{SkipFlow} mechanism also acts as an auxiliary memory. Secondly, modeling relationships between multiple positions allows our model to learn features that represent and approximate textual coherence. In our model, we call this \textit{neural coherence} features. Overall, we present a unified deep learning architecture that generates neural coherence features as it reads in an end-to-end fashion. Our approach demonstrates state-of-the-art performance on the benchmark ASAP dataset, outperforming not only feature engineering baselines but also other deep learning models.
BibTeX
@inproceedings{tay2018skipflow,
  author = {Yi Tay and Minh C. Phan and Luu Anh Tuan and Siu Cheung Hui},
  title = {{SkipFlow: Incorporating Neural Coherence Features for End-to-End Automatic Text Scoring}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  url = {https://openreview.net/forum?id=r1-6wTg_-B},
}
Figure