Abstract
When completing knowledge-intensive tasks, humans sometimes need not just an answer but also a corresponding reference passage for auxiliary reading. Previous methods required obtaining pre-segmented article chunks through additional retrieval models. This paper explores leveraging the parameterized knowledge stored during the pre-training phase of large language models (LLMs) to independently recall reference passage from any starting position. We propose a two-stage framework that simulates the scenario of humans recalling easily forgotten references. Initially, the LLM is prompted to recall document title identifiers to obtain a coarse-grained document set. Then, based on the acquired coarse-grained document set, it recalls fine-grained passage. In the two-stage recall process, we use constrained decoding to ensure that content outside of the stored documents is not generated. To increase speed, we only recall a short prefix in the second stage, then locate its position to retrieve a complete passage. Experiments on KILT knowledge-sensitive tasks have verified that LLMs can independently recall reference passage location in various task forms, and the obtained reference significantly assist downstream tasks.
Introduction
Traditional Retrieval-Augmented Generation (RAG) systems typically rely on external retrieval models like BM25 or DPR to find relevant information. These systems often require "chunking"—pre-segmenting documents into fixed-size blocks—which can limit flexibility and context. If a relevant answer spans multiple chunks or begins mid-segment, the retriever's effectiveness is compromised. Furthermore, maintaining external vector databases and retrieval infrastructure adds significant memory and computational overhead.
This research introduces MindRef, a framework that asks whether LLMs can bypass external chunking and recall references directly from their internal parameters. By mimicking the human memory process—moving from a broad concept (a document title) to a specific detail (a passage)—MindRef enables LLMs to function as independent, character-level retrievers.
Methodology
MindRef utilizes a two-stage retrieval process designed to narrow the search space from a massive corpus to a specific passage location.
- Stage 1: Coarse-Grained Document Recall: The LLM is prompted to generate the title of a relevant document. To prevent hallucinations, the framework employs constrained decoding using a Trie (prefix tree). This ensures the model only generates valid Wikipedia titles.
- Stage 2: Fine-Grained Passage Recall: Once a document is identified, the LLM recalls a specific passage. To maintain accuracy, an FM-index constrains the generation to text that exists within the selected document.
Optimization: Short Prefix Recall and Localization (SPRL)
To address the latency of generating long passages, the authors developed SPRL. Instead of generating a full passage, the LLM generates a short, unique prefix (e.g., 16 tokens). The system then uses the FM-index and the KMP algorithm to locate this "fingerprint" within the document and retrieve the full surrounding context. This approach provides a 4x speedup in inference while maintaining high retrieval accuracy.
Results
The framework was evaluated on the KILT benchmark, covering open-domain QA (NQ, TriviaQA, HotpotQA, ELI5), fact-checking (FEVER), and dialogue (WoW). MindRef, particularly the LLaMA-2-13b variant, demonstrated exceptional performance.
As shown in the results, MindRef achieved an R-Precision of 83.69% on FEVER and 57.63% on WoW, significantly outperforming the fully trained DPR model. Notably, MindRef outperformed unsupervised baselines like BM25 and Contriever across nearly all tasks without requiring task-specific fine-tuning for retrieval. Furthermore, the recalled passages significantly improved the performance of downstream "reading" models, reaching 78.79% accuracy on FEVER.
Discussion
MindRef's success highlights several key insights:
- Hierarchical Retrieval: Breaking down the retrieval task into document-then-passage stages makes internal recall feasible for LLMs by drastically reducing the search space.
- Constrained Decoding as Safety: The use of Tries and FM-indexes effectively transforms a generative model into a deterministic retriever, eliminating the hallucination issues common in LLMs.
- Efficiency: MindRef is highly memory-efficient. Its indexing structures (Trie + FM-index) require only ~8GB of memory, compared to the 60GB+ required for dense retriever embeddings like DPR.
- Natural Context: By allowing retrieval to start at any character position, MindRef provides more natural and contextually relevant passages than fixed-chunk systems.
Conclusion
MindRef demonstrates that LLMs can independently and accurately recall reference passages from their internal parameters. While the model currently faces challenges with "long-tail" knowledge (infrequently seen documents) and requires fine-tuning for knowledge updates, it represents a significant step toward more integrated and efficient retrieval-augmented systems. Future work will focus on instruction tuning for constrained recall and methods for lightweight knowledge injection.
Summary updated