Abstract
CorpusLM is a unified language model that integrates generative retrieval and answer generation into a single, continuous decoding process. By treating document identifiers (DocIDs) as part of the natural language output, it achieves state-of-the-art performance on knowledge-intensive tasks while reducing memory overhead by over 130x compared to traditional retrieval-augmented systems.
Introduction
Large Language Models (LLMs) are notorious for "hallucinating"—generating facts that sound plausible but are entirely false. This is a major hurdle for Knowledge-Intensive (KI) tasks like open-domain question answering or fact-checking.
The standard solution is Retrieval-Augmented Generation (RAG), which fetches relevant documents from an external corpus to ground the LLM's response. However, traditional RAG has two major pain points: a structural disconnect between the retriever and generator, and high resource intensity due to the need for massive vector indices. CorpusLM addresses these issues by unifying retrieval and generation into one cohesive auto-regressive model.
Methods
The authors propose CorpusLM, which unifies retrieval, closed-book generation, and RAG into one auto-regressive model. Key innovations include:
- Unified Greedy Decoding: Instead of separate steps, the model performs a "continuous" generation. For a RAG task, it sequentially decodes: DocIDs → Fine-grained References → Final Answer.
- Ranking-oriented DocID List Generation: CorpusLM is trained to generate a ranked list of DocIDs in natural language, improving its ability to understand document relationships.
- Continuous RAG Strategy: The model is trained to extract specific "references" from documents before answering. It also uses Noise Sampling during training to ensure it doesn't blindly trust retrieved context.
- Auxiliary DocID Understanding: Four unsupervised tasks (e.g., predicting a DocID from a summary) are used to bake corpus knowledge directly into the model's weights.
Experimental Setup
The model was evaluated on the KILT (Knowledge-Intensive Language Tasks) benchmark, which includes 11 datasets across five categories: Fact Checking (FEVER), Entity Linking (AY2, WnWi, WnCw), Slot Filling (T-REx, zsRE), Open Domain QA (NQ, HoPo, TQA, ELI5), and Dialogue (WoW). The external corpus used was a 2019 Wikipedia dump consisting of 5.9 million articles.
Key Results
CorpusLM demonstrates superior performance across the KILT benchmark. In terms of generative retrieval, it consistently outperforms strong baselines like MT-DPR and SimLM.
Key findings include:
- Superior Retrieval and RAG: CorpusLM (Llama2) significantly outperformed existing models. For example, on the FEVER dataset, it achieved a 90.22 score in the RAG setting, surpassing standard RAG (87.70).
- Massive Efficiency Gains: CorpusLM (T5) showed a 139.1-fold decrease in memory footprint compared to traditional RAG because it doesn't need a massive external document index. It also achieved a 2.1-fold reduction in query latency.
- The Model is the Index: By training the model to generate DocIDs, the "index" is effectively stored within the model's parameters, allowing it to use its full linguistic power to "reason" about document relevance.
Conclusion and Future Work
CorpusLM represents a significant step toward unified language modeling for KI tasks. By treating retrieval as a text generation task, it achieves high accuracy and efficiency. Future work will focus on scaling this "index-in-weights" approach to larger corpora and improving the process for dynamic knowledge updates, as adding new documents currently requires model fine-tuning.
Summary updated