Abstract
Reading a document and extracting an answer to a question about its content has attracted substantial attention recently. While most work has focused on the interaction between the question and the document, in this work we evaluate the importance of context when the question and document are processed independently. We take a standard neural architecture for this task, and show that by providing rich contextualized word representations from a large pre-trained language model as well as allowing the model to choose between context-dependent and context-independent word representations, we can obtain dramatic improvements and reach performance comparable to state-of-the-art on the competitive SQuAD dataset.
Introduction
In extractive Reading Comprehension (RC), models are tasked with identifying the specific span of text within a document that answers a given query. Historically, research has prioritized the development of complex "interaction" layers—mechanisms that model the relationship between the question and the document. However, this focus often overlooks how words are represented before they interact.
Standard RC models typically rely on fixed word embeddings like GloVe, which provide a static "dictionary definition" regardless of context. While BiLSTMs are used to incorporate surrounding information, the relatively small size of datasets like SQuAD (Stanford Question Answering Dataset) often prevents models from learning deep contextual nuances from scratch. This paper argues that by leveraging large-scale pre-trained language models, we can provide RC models with a sophisticated understanding of syntax and semantics that they cannot acquire through task-specific training alone.
Methodology: The Re-embedding Module
The core contribution of this work is a re-embedding module placed at the beginning of the neural network. This module utilizes a Highway Layer as a gating mechanism, allowing the model to dynamically balance two types of information:
- Static Word-Type Representations: Fixed embeddings that capture the general meaning of a word.
- Contextualized Representations: Representations that change based on the word's specific usage in a sentence.
Token Re-embedding (TR) and LM Augmentation
The authors explore two ways to generate the contextual component. In the TR (Token Re-embedding) version, the context is learned during training via a BiLSTM over the input. In the more powerful TR+LM (Language Model Augmentation) version, the model incorporates hidden states from a massive Language Model pre-trained on the "One Billion Word Benchmark." By using the LM as a fixed encoder, the model is "gifted" a rich understanding of context without the need for additional labeled RC data.
Key Insights: Context for Rare Words
A significant finding of this research is how the gating mechanism behaves relative to word frequency. By analyzing the gate activations, the authors discovered that the model naturally learns to rely more heavily on context for rare words, while favoring static embeddings for common words.
This behavior mirrors human linguistic processing: when encountering an unfamiliar word, we use the surrounding text to infer its meaning. For common words, the static "dictionary" definition is often sufficient.
Experimental Results
The addition of the contextualized re-embedding module to the RaSoR (Recurrent Span Representations) base model yielded substantial performance gains on the SQuAD benchmark.
As shown in the results, the full TR+LM(L1) model reached 77.0% EM and 84.0% F1, a dramatic improvement over the 70.6% EM of the base model. Interestingly, the authors found that using the first layer (L1) of the pre-trained LM outperformed the second layer (L2), suggesting that lower-level contextual features like syntax are particularly vital for reading comprehension.
Robustness and Future Work
Beyond standard benchmarks, the model demonstrated superior robustness on Adversarial SQuAD, where distractor sentences are added to confuse the system. By having a stronger grasp of context, the model was less likely to be misled by keyword-matching distractors compared to more complex co-attention models.
While the current approach uses a fixed LM, future work—which eventually led to architectures like BERT—suggests that fine-tuning the language model specifically for the RC task could provide even greater benefits. Additionally, combining these rich representations with more advanced interaction layers remains a promising avenue for further performance gains.
Summary updated