TL;DR
This paper introduces a method to improve language models' understanding of long documents by allowing them to access "ancient history" – text outside their immediate context window. By training a separate "selector" model, they can intelligently pull relevant snippets from this distant past into the model's view, significantly reducing prediction errors without altering the core language model itself.
What Problem Does This Paper Address?
Modern language models (LMs) are getting better at handling longer sequences of text, with context windows expanding to thousands of tokens. However, many real-world applications, like understanding scientific papers or legal documents, involve texts that are tens of thousands of tokens long. This creates a challenge: how can LMs effectively leverage information that falls outside their fixed context window, especially when the parameters of these large models are often kept secret and cannot be easily retrained? This paper tackles this by finding a way to "reach back" into the document's past, beyond the immediate context.
Methods
The core idea is to augment the LM's context window by incorporating relevant snippets from "ancient history" – text that precedes the current context window. They achieve this by training an auxiliary "selection function." This function takes the current context and the ancient history as input and learns to identify and select the most predictive spans from the ancient history. These selected spans are then inserted into the LM's context window, replacing less useful parts.
Crucially, this approach works with off-the-shelf, pretrained LMs without any modification to their parameters. The selection function is trained separately. Its objective is to maximize the predicted log ratio of the likelihood of future words given the selected span versus the likelihood given only the necessary prefix (the part of the history that is within the context window). They train this selection function using a regression approach, minimizing the squared error. For inference, they divide the context window into two parts: a "necessary prefix" and a "selected history" section. The selection function then fills this "selected history" section with top-scoring spans from the ancient history.
Datasets
- Wikitext-2: Used for training the selection function and for in-domain experiments with GPT-2 models.
- S2ORC corpus: A collection of 1,451 scientific papers, used for cross-domain experiments. Documents in this corpus are significantly longer than Wikipedia articles.
Key Insights and Ideas
- Ancient history is valuable: Information far outside the immediate context window can be highly predictive of future text.
- Non-contiguous context works: LMs can benefit from non-contiguous context, meaning they don't need all the information to be right next to each other.
- Transferability of selection: A selection function trained on one domain (like Wikipedia) can perform well on a different domain (like scientific papers), suggesting a generalizable understanding of "useful" historical context.
- Auxiliary models are powerful: A relatively simple auxiliary model can guide a much larger LM without needing to retrain the large model itself.
- "Prompting" for documents: The selected history can be viewed as a form of "prompting" for document-level tasks, especially relevant for large, inaccessible models.
Key Results
The paper reports perplexity reductions on both Wikipedia and scientific text datasets:
- GPT-2 Large on Wikitext-2: A 0.9 absolute perplexity reduction compared to the baseline.
- GPT-2 Small on Wikitext-2: A 2.48 absolute perplexity reduction.
- Cross-domain (Wikitext-2 trained selector on S2ORC): A 2.59 absolute perplexity reduction.
- Overall, they observed a 7% perplexity reduction on Wikipedia articles and a 12% perplexity reduction on scientific texts.
Limitations and Future Work
- Computational cost: The current selection function architecture, while effective, is computationally expensive. Future work could explore lighter architectures or more efficient ways to compute representations.
- Fixed span size: The current method uses fixed-size spans. Future work could explore variable-length spans or dynamically determining the total amount of selected content per document or timestep.
- Oracle performance: While their learned selection function performs very close to an oracle (which has perfect knowledge), there's still a small gap, suggesting room for improvement.
- Interpretability: While the selected spans offer some interpretability, future work could explore methods for selecting out-of-document contexts more explicitly.
- Generalization to other tasks: The paper suggests exploring if this reward-based training for auxiliary models can benefit other NLP tasks beyond language modeling.
Summary updated