StackReader: An RNN-Free Reading Comprehension Model

Yibo Jiang, Zhou Zhao

Abstract

StackReader is a reading comprehension model that replaces recurrent neural networks (RNNs) with an architecture based on multi-head attention and gated convolutional neural networks (CNNs). By removing the sequential processing bottleneck of RNNs, StackReader achieves performance comparable to established baselines like BiDAF while reducing training time by approximately 50%.

Introduction

In Machine Reading Comprehension (MRC), systems are designed to answer questions based on provided text passages. Traditionally, this task has been dominated by Recurrent Neural Networks (RNNs) such as LSTMs and GRUs. However, RNNs suffer from two primary limitations: a sequential bottleneck that hinders GPU parallelization and a tendency to lose global context over long distances.

Inspired by the success of the Transformer and ConvS2S in machine translation, StackReader demonstrates that a high-performing MRC model can be constructed without any recurrent units, leveraging attention and convolutions for efficiency.

The StackReader Architecture

The StackReader architecture is composed of three primary layers designed to maximize parallelization and context capture.

StackReader architecture diagram showing the stack of self-attention and interactive-attention layers.
  1. Embedding Layer: This layer utilizes a hybrid approach, combining word-level embeddings (pre-trained GloVe vectors) with character-level embeddings generated via a CNN. This dual approach ensures the model captures both semantic meaning and handles out-of-vocabulary terms effectively.
  2. Modeling and Attention Layer: This core component intertwines contextual modeling and attention. It employs multi-head attention to process different parts of the text simultaneously and alternates between interactive attention (query-to-context) and self-attention (context-to-context). To replicate the selective filtering of LSTMs without the sequential cost, Gated Linear Units (GLUs) are integrated after the attention layers.
  3. Output Layer: The model predicts answer spans using Maxout networks and an "Attention-over-Attention" (AoA) mechanism to refine the probability distributions for the start and end indices.

Experimental Setup

The model was evaluated on the Stanford Question Answering Dataset (SQuAD), a benchmark containing over 100,000 question-answer pairs. In SQuAD, the answer is always a segment of text within the provided passage.

Results and Discussion

StackReader was compared against the Bidirectional Attention Flow (BiDAF) model on the SQuAD development set. The results highlight a significant efficiency gain with a manageable trade-off in accuracy.

Comparison of StackReader and BiDAF performance on SQuAD, showing EM, F1, and training time.

As shown in the table, StackReader reduced training time from 16.5 hours to approximately 8 hours. While its Exact Match (EM) and F1 scores (65.86 and 75.70, respectively) are slightly lower than BiDAF's, the nearly 50% reduction in training time makes it a compelling alternative for resource-constrained environments.

The inclusion of Gated CNNs proved crucial, providing the "memory management" capability typically associated with LSTMs while maintaining the parallel processing advantages of the attention-based architecture.

Conclusion and Future Work

StackReader proves that recurrence is not a prerequisite for effective reading comprehension. While a small performance gap remains compared to top RNN-based models, the efficiency gains are substantial. Future work involves exploring binarized neural networks—using 1-bit weights and activations—to further reduce the model's memory footprint and enhance inference speed for mobile and edge applications.

Summary updated

Abstract

Machine comprehension of text is the problem to answer a query based on a given context. Many existing systems use RNN-based units for contextual modeling linked with some attention mechanisms. In this paper, however, we propose StackReader, an end-to-end neural network model, to solve this problem, without recurrent neural network (RNN) units and its variants. This simple model is based solely on attention mechanism and gated convolutional neural network. Experiments on SQuAD have shown to have relatively high accuracy with a significant decrease in training time.
BibTeX
@inproceedings{jiang2018stackreader,
  author = {Yibo Jiang and Zhou Zhao},
  title = {{StackReader: An RNN-Free Reading Comprehension Model}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12169},
}
Figure