Abstract
Traditional Large Language Models (LLMs) rely on parametric memory—knowledge stored within their weights—which often leads to factual hallucinations and an inability to access up-to-date or private information. While Retrieval-Augmented Language Modeling (RALM) addresses these issues by incorporating external documents, existing methods typically require architectural modifications and expensive retraining. This paper explores In-Context RALM, a "zero-effort" approach that leaves the language model (LM) frozen and simply prepends retrieved documents to the input context. Our findings demonstrate that this simple integration allows smaller models to match or exceed the performance of models ten times their size, providing a significant "virtual" increase in model scale.
Introduction
As LLMs scale, the cost of updating their internal knowledge becomes prohibitive. Standard RALM systems, such as DeepMind’s RETRO, mitigate this by conditioning the model on external data, but they necessitate specialized "document reading" modules and retraining. This creates a barrier for researchers using massive frozen models or developers restricted by API-based access.
In-Context RALM asks a fundamental question: Can we achieve the benefits of retrieval-augmentation without modifying the model's weights? By treating the retriever as a plug-and-play component that feeds into the LM's existing context window, we can ground the model in external evidence using off-the-shelf components.
Methodology
The In-Context RALM framework operates through a straightforward grounding mechanism. For a given input prefix, a retriever identifies relevant documents from a corpus (such as Wikipedia), which are then prepended to the input before being processed by the frozen LM.
Optimization Parameters
To make this process efficient and effective, the authors investigated two critical hyperparameters:
- Retrieval Stride ($s$): To avoid the prohibitive latency of retrieving for every single token, the system retrieves every $s$ tokens. Experiments showed that while more frequent retrieval (lower $s$) improves performance, a stride of $s=4$ provides an optimal balance between speed and accuracy.
- Retrieval Query Length ($\ell$): The length of the query used to fetch documents is vital. If the query is too short, it lacks sufficient context; if it is too long, the specific local information needed for the next-token prediction is diluted.
Reranking Strategies
To further refine the quality of retrieved context, the authors introduced two reranking methods:
- Zero-shot Reranking: A small, off-the-shelf LM scores documents based on how well they help predict the immediate prefix.
- Predictive Reranking: A dedicated model (RoBERTa) is trained to predict which document will maximize the likelihood of the LM correctly generating the upcoming text.
Experimental Evaluation
The framework was evaluated across diverse benchmarks, including WikiText-103, RealNews, and subsets of The Pile (ArXiv, Stack Exchange, and FreeLaw), as well as Open-Domain Question Answering (ODQA) tasks like Natural Questions (NQ) and TriviaQA. The study utilized a wide range of models, including GPT-2, GPT-Neo, OPT, and LLaMA, with parameter counts ranging from 110M to 66B.
Key Insights
- Retrieval as a Force Multiplier: In-Context RALM acts as a significant efficiency booster. For instance, an OPT-6.7B model using this method matched the perplexity of a 66B parameter model, effectively providing a 10x increase in "virtual" scale.
- The Efficacy of Sparse Retrieval: Surprisingly, the keyword-based BM25 retriever consistently outperformed modern neural dense retrievers like Contriever for language modeling tasks. This suggests that lexical overlap remains a more reliable signal for general text completion than abstract semantic embeddings.
- Small Rerankers are Sufficient: The study found that a 110M parameter GPT-2 model could effectively rerank documents for a 1.5B parameter model with negligible performance loss compared to the larger model reranking for itself.
Results
The empirical results highlight the dramatic gains possible through in-context grounding:
- WikiText-103: A GPT-2 345M model with In-Context RALM achieved a perplexity of 19.7, surpassing the 20.0 perplexity of a much larger GPT-2 1.5B model without retrieval.
- Open-Domain QA: On the Natural Questions dataset, LLaMA-13B’s Exact Match (EM) score improved from 12.0% in a closed-book setting to 31.0% in an open-book setting simply by prepending two retrieved documents.
- Reranking Gains: The predictive reranker lowered GPT-2 XL's perplexity on WikiText-103 from 16.6 (standard BM25) to 15.4.
Discussion and Limitations
While In-Context RALM offers a low-barrier path to grounding LLMs, it faces several constraints:
- Context Window Limits: Prepending documents consumes the LM's limited context window. While newer models with 32k+ token limits alleviate this, it remains a bottleneck for standard architectures.
- Latency: Retrieving every few tokens introduces computational overhead, making real-time deployment challenging without significant optimization.
- Context Management: Current experiments focused primarily on single-document prepending. Future work is needed to explore how multiple documents can be managed effectively without overwhelming the model's attention mechanism.
Conclusion
In-Context RALM demonstrates that architectural changes are not a prerequisite for effective retrieval augmentation. By leveraging the existing in-context learning capabilities of frozen LMs, researchers and developers can significantly enhance model performance, factual accuracy, and scalability with minimal engineering effort.
Summary updated