Generate rather than Retrieve: Large Language Models are Strong Context Generators

Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, Meng Jiang

This paper, presented at ICLR 2023, challenges the long-standing "retrieve-then-read" paradigm in NLP. Instead of searching through a massive external corpus like Wikipedia to find answers, the authors propose GENREAD: a "generate-then-read" pipeline where a Large Language Model (LLM) first generates its own contextual documents from its internal parameters and then reads them to produce a final answer.

TL;DR

GENREAD replaces traditional document retrievers with LLM generators, proving that LLMs can "retrieve" knowledge from their own weights more effectively than external search engines for many tasks. It achieves state-of-the-art results on open-domain QA benchmarks like TriviaQA and WebQ without accessing any external knowledge source during inference.

What Problem Does This Paper Address?

For "knowledge-intensive" tasks—like answering "Who won the first Nobel Prize in Physics?"—standard AI models usually rely on a retrieve-then-read pipeline. They use a retriever (like DPR) to find relevant snippets in Wikipedia and a reader to extract the answer.

However, retrieval has three major flaws:

  1. Noise: Documents are often pre-chunked into fixed sizes (e.g., 100 words), which might include irrelevant text.
  2. Shallow Interaction: Most retrievers use "two-tower" models that don't allow the question and the document to "talk" to each other deeply during the search.
  3. Parameter Bottlenecks: Retrievers are often small models limited by the size of their embedding vectors, preventing them from using the vast "world knowledge" stored in massive LLMs.

Methods

The authors introduce GENREAD (Generate-then-Read), which operates in two distinct steps:

  1. Generate: Given a question, the LLM is prompted to "Generate a background document from Wikipedia to answer the given question."
  2. Read: The same (or a different) model reads the generated document to produce the final answer.

Key Innovations:

  • Clustering-Based Prompting: To improve "recall" (the chance that the answer is actually in the text), the model needs to generate diverse documents. Simply sampling multiple times often leads to repetitive text. The authors clustered a training set of question-document pairs into $K$ groups. By sampling few-shot examples from different clusters, they "bias" the LLM to generate documents from different perspectives, significantly increasing knowledge coverage.
Clustering-based prompting pipeline
  • Zero-Shot vs. Supervised: The method works both in a zero-shot setting (using InstructGPT for both steps) and a supervised setting (where a smaller "Reader" model like FiD is fine-tuned specifically on these generated documents).

Datasets

The authors evaluated GENREAD across three categories of knowledge-intensive tasks:

  • Open-Domain QA: Natural Questions (NQ), TriviaQA, and WebQuestions (WebQ).
  • Fact Checking: FEVER and Fool Me Twice (FM2).
  • Dialogue Systems: Wizard of Wikipedia (WoW).

Key Insights and Ideas

  • LLMs as Internal Retrievers: The authors argue that the pre-training objective of LLMs (predicting the next token) is fundamentally similar to document generation. Therefore, an LLM is naturally better at "writing" the context it needs than a separate model is at "finding" it.
  • Deep Token-Level Interaction: Because the LLM generates the document conditioned on the question, it performs deep token-level cross-attention. This results in context that is much more specific to the query than a generic Wikipedia snippet.
  • The "Aha" Moment: Surprisingly, generated documents often contain the correct answer more frequently than the top-1 retrieved document from a traditional system. The LLM essentially "distills" its internal knowledge into a readable format for the reader.
  • Complementarity: While GENREAD is powerful alone, the best results come from a hybrid approach: merging retrieved documents with generated ones. This combines the "groundedness" of real text with the "reasoning" of generated text.

Key Results

  • TriviaQA & WebQ: GENREAD achieved 71.6 and 54.4 Exact Match (EM) scores, respectively. This outperformed the state-of-the-art retrieval-based pipeline (DPR-FiD) by +4.0 and +3.9 points, despite GENREAD not "seeing" any external documents.
Recall@K performance on TriviaQA and WebQ
  • Zero-Shot Boost: In a zero-shot setting, GENREAD improved InstructGPT's performance by +6.9 points on average across QA tasks compared to just asking the model for the answer directly.
  • Hybrid Superiority: By merging retrieved and generated documents, the authors reached an average score of 61.5 across QA benchmarks, a significant jump over using either method in isolation.

Limitations and Future Work

  • Temporality: LLMs are frozen in time. If a question concerns an event that happened after the model's training cutoff, GENREAD will likely hallucinate or fail, whereas a retriever can find the latest news.
  • Hallucination: While the "Reader" helps filter out some errors, the "Generator" can still produce factually incorrect documents that look convincing.
  • Inference Cost: Generating 10+ documents per question is computationally much more expensive and slower than performing a vector search in a database.
  • Future Direction: The authors suggest exploring how to efficiently update the LLM's internal knowledge without full retraining to solve the temporality issue.

Summary updated

Abstract

Knowledge-intensive tasks, such as open-domain question answering (QA), require access to a large amount of world or domain knowledge. A common approach for knowledge-intensive tasks is to employ a retrieve-then-read pipeline that first retrieves a handful of relevant contextual documents from an external corpus such as Wikipedia and then predicts an answer conditioned on the retrieved documents. In this paper, we present a novel perspective for solving knowledge-intensive tasks by replacing document retrievers with large language model generators. We call our method generate-then-read (GenRead), which first prompts a large language model to generate contextual documents based on a given question, and then reads the generated documents to produce the final answer. Furthermore, we propose a novel clustering-based prompting method that selects distinct prompts, in order to generate diverse documents that cover different perspectives, leading to better recall over acceptable answers. We conduct extensive experiments on three different knowledge-intensive tasks, including open-domain QA, fact checking, and dialogue system. Notably, GenRead achieves 71.6 and 54.4 exact match scores on TriviaQA and WebQ, significantly outperforming the state-of-the-art retrieve-then-read pipeline DPR-FiD by +4.0 and +3.9, without retrieving any documents from any external knowledge source. Lastly, we demonstrate the model performance can be further improved by combining retrieval and generation. Our code and generated documents can be found at https://github.com/wyu97/GenRead.
BibTeX
@inproceedings{yu2023generate,
  author = {Wenhao Yu and Dan Iter and Shuohang Wang and Yichong Xu and Mingxuan Ju and Soumya Sanyal and Chenguang Zhu and Michael Zeng and Meng Jiang},
  title = {{Generate rather than Retrieve: Large Language Models are Strong Context Generators}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=fB0hRu9GZUS},
}
Figure