Introduction
In modern information retrieval, Large Language Models (LLMs) are typically reserved for the second-stage re-ranking of documents due to their high computational cost. First-stage retrieval, which narrows down millions of documents to a manageable candidate set, usually relies on simpler models like BM25 or bi-encoders trained via expensive unsupervised contrastive distillation. PromptReps introduces a zero-shot alternative that leverages generative LLMs directly for first-stage retrieval without any additional training or fine-tuning.
Methodology: Representation through Prompting
The core mechanism of PromptReps is "representation through prompting." By providing a specific instruction—such as "Use one word to represent the passage in a retrieval task"—the model is forced to compress the semantic essence of a document into a single prediction step.
Dual Representations
PromptReps extracts two distinct representations from a single forward pass of the LLM:
- Dense Representation: The hidden state of the final token in the prompt is used as a high-dimensional semantic vector.
- Sparse Representation: The logits (probability distribution) for the next predicted token are extracted. This distribution is sparsified by retaining weights only for tokens that appear in the original document, creating a weighted bag-of-words.
These representations are stored in an Approximate Nearest Neighbor (ANN) index and a traditional inverted index, respectively. Retrieval is performed by linearly interpolating the scores from both indices.
Experimental Evaluation
The authors evaluated PromptReps on major benchmarks including MSMARCO, TREC Deep Learning (2019 & 2020), and the BEIR suite. The method was tested across various LLM architectures, including Llama 3 (8B and 70B), Mistral-7B, and Phi-3-mini.
Results and Discussion
The experiments demonstrate that the hybrid combination of dense and sparse representations significantly outperforms either method in isolation.
Key Findings
- The Hybrid Advantage: While dense embeddings from LLMs often struggle in zero-shot settings, the sparse component provides critical keyword robustness. On the BEIR benchmark, PromptReps (Llama 3-70B) achieved an average nDCG@10 of 45.97, surpassing the BM25 baseline (43.70).
- Scaling Effects: Retrieval performance improves as the underlying LLM scales. Moving from Llama 3-8B to 70B resulted in substantial gains, indicating that stronger generative capabilities translate to better document representations.
- Zero-Shot Efficiency: PromptReps achieves results competitive with state-of-the-art supervised models like E5-mistral without requiring the massive GPU resources typically needed for contrastive training.
Conclusion
PromptReps demonstrates that generative LLMs can be effectively repurposed as first-stage retrievers through clever prompting. By extracting both dense and sparse signals, the method achieves high performance in zero-shot scenarios. While query latency remains a challenge compared to smaller encoders, the elimination of the training phase and the ability to leverage scaling laws make PromptReps a compelling new direction for information retrieval.
Summary updated