Retrieval-based Language Models and Applications

Akari Asai, Sewon Min, Zexuan Zhong, Danqi ChenACL 2023 · DOI

TL;DR

This tutorial proposal outlines a comprehensive overview of retrieval-based language models (LMs), which augment traditional LMs with external knowledge retrieval. It addresses the limitations of parametric LMs in encoding vast world knowledge, adapting over time, and avoiding hallucinations, by exploring architectures, training methods, and applications of these more efficient and adaptable models.

What Problem Does This Paper Address?

Traditional large language models (LLMs) like GPT-3 and PaLM excel at many NLP tasks but face significant challenges. Encoding the entirety of human knowledge into their parameters requires immense computational resources and leads to models that struggle with "long-tail" knowledge (rare facts or events). Furthermore, these parametric models are static; they cannot easily update their knowledge without costly retraining and are prone to generating incorrect information (hallucinations) or even leaking private training data. This tutorial proposal aims to bridge this gap by focusing on retrieval-based LMs, which integrate a non-parametric datastore (like a collection of text chunks) with their parametric counterparts. This approach offers a path to more efficient, adaptable, and verifiable language models.

Methods

The tutorial proposes to cover retrieval-based LMs through a structured approach:

  • Taxonomy of Architectures: The authors will categorize retrieval-based LMs based on two key dimensions:
  • Granularity of Retrieved Units: This includes models that retrieve individual tokens, entity mentions, or larger chunks of text.
  • Integration of Retrieved Information: How the retrieved context is incorporated into the LM, whether by concatenating it with the input, integrating it into intermediate layers of the Transformer, or interpolating its distribution with the LM's own.
  • Scalable Learning Algorithms: Addressing the challenge of training with massive retrieval datastores, the tutorial will discuss:
  • Pipelined Training: Training retrieval components and LMs separately.
  • In-batch Approximations: Using subsets of the corpus during joint training.
  • Joint Training with Asynchronous Updates: Continuously updating the retrieval index while training the LM.
  • Downstream Adaptations: The tutorial will explore how these models are adapted to specific tasks, including zero-shot/few-shot prompting and fine-tuning, and their application in areas like dialogue and semantic parsing.
  • Extensions: The proposal also includes coverage of multilingual, multimodal, and code-related retrieval-based LMs.

Datasets

Not explicitly detailed in the proposal, but the context implies the use of large text corpora for the retrieval datastores, and standard NLP datasets for evaluating downstream task performance.

Key Insights and Ideas

  • Efficiency and Adaptability: Retrieval-based LMs offer a compelling alternative to ever-larger parametric models, providing a more energy-efficient and cost-effective way to incorporate and update world knowledge.
  • Verifiability: The ability to retrieve and cite external sources can significantly improve the trustworthiness and verifiability of LM outputs, a crucial step towards reliable AI.
  • Modular Knowledge: By separating knowledge into a retrievable datastore, these models can be updated more easily and potentially avoid some of the privacy concerns associated with embedding sensitive data directly into model parameters.
  • Unified Framework: The tutorial aims to provide a coherent overview, unifying disparate research efforts in retrieval-based LMs under a common taxonomy and set of principles.

Key Results

The proposal does not present specific quantitative results as it is a tutorial outline. However, it highlights that retrieval-based LMs can "outperform LMs without retrieval by a large margin with much fewer parameters" and can "update their knowledge by replacing their retrieval corpora."

Limitations and Future Work

The authors acknowledge that retrieval-based LMs, while reducing hallucinations, can still produce incorrect or nonsensical outputs. A significant ethical consideration is the potential for retrieved data to leak private information if the datastore is built on sensitive data. Future directions discussed include:

  • Improving the scalability of retrieval-based LMs without sacrificing performance.
  • Determining the optimal use cases for retrieval-based LMs in the context of rapidly evolving LLMs.
  • Enabling applications of retrieval-based LMs across a wider range of domains.

Summary updated

BibTeX
@inproceedings{asai2023retrieval,
  author = {Akari Asai and Sewon Min and Zexuan Zhong and Danqi Chen},
  title = {{Retrieval-based Language Models and Applications}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-tutorials.6},
}
Figure