Abstract
StackReader is a reading comprehension model that replaces recurrent neural networks (RNNs) with an architecture based on multi-head attention and gated convolutional neural networks (CNNs). By removing the sequential processing bottleneck of RNNs, StackReader achieves performance comparable to established baselines like BiDAF while reducing training time by approximately 50%.
Introduction
In Machine Reading Comprehension (MRC), systems are designed to answer questions based on provided text passages. Traditionally, this task has been dominated by Recurrent Neural Networks (RNNs) such as LSTMs and GRUs. However, RNNs suffer from two primary limitations: a sequential bottleneck that hinders GPU parallelization and a tendency to lose global context over long distances.
Inspired by the success of the Transformer and ConvS2S in machine translation, StackReader demonstrates that a high-performing MRC model can be constructed without any recurrent units, leveraging attention and convolutions for efficiency.
The StackReader Architecture
The StackReader architecture is composed of three primary layers designed to maximize parallelization and context capture.
- Embedding Layer: This layer utilizes a hybrid approach, combining word-level embeddings (pre-trained GloVe vectors) with character-level embeddings generated via a CNN. This dual approach ensures the model captures both semantic meaning and handles out-of-vocabulary terms effectively.
- Modeling and Attention Layer: This core component intertwines contextual modeling and attention. It employs multi-head attention to process different parts of the text simultaneously and alternates between interactive attention (query-to-context) and self-attention (context-to-context). To replicate the selective filtering of LSTMs without the sequential cost, Gated Linear Units (GLUs) are integrated after the attention layers.
- Output Layer: The model predicts answer spans using Maxout networks and an "Attention-over-Attention" (AoA) mechanism to refine the probability distributions for the start and end indices.
Experimental Setup
The model was evaluated on the Stanford Question Answering Dataset (SQuAD), a benchmark containing over 100,000 question-answer pairs. In SQuAD, the answer is always a segment of text within the provided passage.
Results and Discussion
StackReader was compared against the Bidirectional Attention Flow (BiDAF) model on the SQuAD development set. The results highlight a significant efficiency gain with a manageable trade-off in accuracy.
As shown in the table, StackReader reduced training time from 16.5 hours to approximately 8 hours. While its Exact Match (EM) and F1 scores (65.86 and 75.70, respectively) are slightly lower than BiDAF's, the nearly 50% reduction in training time makes it a compelling alternative for resource-constrained environments.
The inclusion of Gated CNNs proved crucial, providing the "memory management" capability typically associated with LSTMs while maintaining the parallel processing advantages of the attention-based architecture.
Conclusion and Future Work
StackReader proves that recurrence is not a prerequisite for effective reading comprehension. While a small performance gap remains compared to top RNN-based models, the efficiency gains are substantial. Future work involves exploring binarized neural networks—using 1-bit weights and activations—to further reduce the model's memory footprint and enhance inference speed for mobile and edge applications.
Summary updated