Abstract
Reading Comprehension (RC) models often struggle to utilize context effectively because they are typically trained on relatively small datasets. This paper introduces a gating mechanism that dynamically chooses between contextual and non-contextual word representations. By augmenting these representations with features from a large pre-trained language model, the authors achieve state-of-the-art results on the SQuAD benchmark and significantly improve model robustness against adversarial distractors.
Introduction
In extractive Reading Comprehension, a model must identify the specific span of text within a passage that answers a given question. While modern models utilize BiLSTMs to capture sequence information, they often become "lazy." Because RC datasets like SQuAD are small compared to the complexity of natural language, models tend to rely on simple keyword matching rather than deep contextual understanding.
The core challenge lies in the nature of standard word embeddings (e.g., GloVe), which are static; the representation for a word like "bank" remains identical regardless of whether the context refers to a river or a financial institution. The authors hypothesize that the training signal in standard RC tasks is insufficient to force models to utilize context deeply, leading to a reliance on these static word-type representations.
Methodology
The authors propose a "re-embedding" module designed to be prepended to existing RC architectures, using the RaSoR model as their baseline. The architecture consists of two primary innovations:
1. The Gating Mechanism
Instead of directly feeding embeddings into a sequence encoder, the model uses a Highway layer to calculate a gate value ($g_t$) for every token. This gate allows the model to dynamically decide the balance between the static word-type representation (non-contextual) and a representation derived from the surrounding sequence (contextual).
2. Token Re-embedding (TR) and LM Augmentation
- RNN-based TR: The contextual term is derived from the hidden states of a BiLSTM trained specifically on the RC task.
- LM-augmented TR (TR+LM): To forcibly facilitate context utilization, the authors leverage internal hidden states from a massive, pre-trained "One Billion Word" Language Model as fixed features. This provides the RC model with rich linguistic context that it does not have to learn from the limited RC training data alone.
Key Insights: Rare Words and Context
By inspecting the gating mechanism, the authors discovered that the model assigns lower gate values to rare words. This indicates that the model "recognizes" when its static embedding for a rare word is likely insufficient, choosing instead to rely more heavily on surrounding context to determine meaning.
Experimental Results
The proposed modifications yielded significant improvements across multiple benchmarks. Adding the basic re-embedding (TR) to the RaSoR model improved the F1 score on the SQuAD development set from 78.7 to 82.5.
Performance and Robustness
- State-of-the-Art: The full TR + LM(L1) model reached 77.0 EM (Exact Match) and 84.0 F1 on the SQuAD development set. At the time of publication, it ranked second in EM among published single-model results.
- Layer Depth: Interestingly, using the first layer of the pre-trained LM (L1) outperformed the second layer (L2). The authors suggest L1 captures more general representational context, while L2 may be too specialized for next-word prediction.
- Adversarial Robustness: On the Adversarial SQuAD dataset, the LM-augmented model significantly outperformed the base model (47.0 F1 vs 39.5 F1). This suggests that better contextual representations help the model ignore "distractor" sentences that share keywords with the question but lack the actual answer.
Conclusion and Future Work
This work demonstrates that the "context gap" in RC models can be bridged by leveraging external linguistic knowledge from pre-trained language models. While the current approach uses the LM as a fixed feature extractor, the authors note that fine-tuning the LM specifically for the RC task—a strategy later popularized by models like BERT—is a promising future direction. Additionally, while this paper focuses on better representations, future work may involve combining these improvements with more sophisticated interaction layers between the question and the document.
Summary updated