Retrieval Augmented Language Model Pre-Training

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, Ming-Wei ChangICML 2020 · OpenReview

TL;DR

REALM transforms language models from "closed-book" memorizers into "open-book" researchers by integrating a neural retriever directly into the pre-training process. By learning to pull relevant documents from a corpus of millions (like Wikipedia) to fill in masked blanks, REALM achieves state-of-the-art performance on Open-QA tasks while being 30x smaller than massive models like T5-11B.

The Problem: Implicit Knowledge Storage

Standard language models (LMs) like BERT or GPT store world knowledge implicitly within their network parameters. While effective, this approach has three major drawbacks:

  1. Scaling Issues: To "know" more facts, the network must get exponentially larger, which is computationally expensive and environmentally taxing.
  2. Lack of Interpretability: It is nearly impossible to determine where a specific fact is stored within the weights or why a model generated a particular answer.
  3. Static Knowledge: Updating the model’s knowledge requires expensive retraining of the entire network.

The authors of REALM propose a more modular approach: instead of forcing the model to memorize everything in its parameters, they teach it to retrieve and attend to external documents during both training and inference.

Methods: Retrieve-Then-Predict

The core of REALM is a generative process that splits the task into two distinct steps. Given an input $x$ (such as a sentence with a missing word), the model operates as follows:

  1. Retrieve: A Neural Knowledge Retriever uses a dense inner product model to find the most relevant documents $z$ from a massive corpus $\mathcal{Z}$.
  2. Predict: A Knowledge-Augmented Encoder takes both the original input $x$ and the retrieved document $z$ to predict the missing tokens $y$.
REALM pre-training architecture

The "Secret Sauce": Asynchronous MIPS

The biggest technical hurdle in this architecture is that the retriever must consider millions of documents for every step. To make this differentiable and efficient, the authors use Maximum Inner Product Search (MIPS). However, as the retriever's parameters update during training, the document embeddings change, making the search index "stale."

REALM solves this by running two jobs in parallel: a trainer that updates parameters and an index builder that asynchronously re-indexes the entire corpus every few hundred steps. This ensures the retriever always has relatively fresh embeddings to search against without bottlenecking the training loop.

Inductive Biases

To force the model to actually rely on the retriever rather than just its internal parameters, the authors introduced clever training strategies:

  • Salient Span Masking: Instead of masking random words (like "the" or "is"), they specifically mask named entities and dates (e.g., "United Kingdom" or "1969") that require explicit world knowledge to predict.
  • Inverse Cloze Task (ICT) Initialization: To avoid a "cold start" where the retriever initially pulls irrelevant junk, they pre-train the retriever to find the document that a specific sentence originally came from.

Key Insights

  • End-to-End Retrieval: Unlike previous systems that used fixed search engines (like BM25), REALM’s retriever is trained via backpropagation. It learns what to search for based purely on the language modeling objective.
  • Efficiency via Modularity: Explicit retrieval is a far more efficient way to handle knowledge than raw parameter scaling. A REALM model with 330 million parameters can outperform models exponentially larger.
  • Interpretability: REALM provides "provenance." When the model answers a question, users can see exactly which Wikipedia passage it used to find that information, making hallucination easier to catch.

Key Results

REALM was evaluated on Open-Domain QA benchmarks including NaturalQuestions-Open (NQ), WebQuestions (WQ), and CuratedTrec (CT).

Open-Domain QA benchmark results
  • State-of-the-Art in Open-QA: REALM outperformed all previous models on the three benchmarks by 4-16% absolute accuracy.
  • David vs. Goliath: On NaturalQuestions-Open, REALM (40.4% accuracy) beat the massive T5-11B model (34.5%) despite being a fraction of the size (330M parameters vs. 11B parameters).
  • Ablation Success: The authors demonstrated that "Salient Span Masking" was critical; without it, the model often relied on local context rather than utilizing the retriever effectively.

Limitations and Future Work

  • Computational Overhead: While the final model is highly efficient during inference, the pre-training process requires significant infrastructure to handle the asynchronous MIPS indexing.
  • Unstructured Text Only: Currently, REALM only retrieves from raw text. Future work could extend this to structured data (like knowledge graphs) or multi-modal data (images/video).
  • Language Scope: The current work focuses entirely on English. Extending this architecture to multi-lingual retrieval remains an open challenge for the community.

Summary updated

Abstract

Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the p...
BibTeX
@inproceedings{guu2020retrieval,
  author = {Kelvin Guu and Kenton Lee and Zora Tung and Panupong Pasupat and Ming-Wei Chang},
  title = {{Retrieval Augmented Language Model Pre-Training}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=kt_63AlKRQH},
}
Figure