PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval

Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, Guido Zuccon

Introduction

In modern information retrieval, Large Language Models (LLMs) are typically reserved for the second-stage re-ranking of documents due to their high computational cost. First-stage retrieval, which narrows down millions of documents to a manageable candidate set, usually relies on simpler models like BM25 or bi-encoders trained via expensive unsupervised contrastive distillation. PromptReps introduces a zero-shot alternative that leverages generative LLMs directly for first-stage retrieval without any additional training or fine-tuning.

Methodology: Representation through Prompting

The core mechanism of PromptReps is "representation through prompting." By providing a specific instruction—such as "Use one word to represent the passage in a retrieval task"—the model is forced to compress the semantic essence of a document into a single prediction step.

Dual Representations

PromptReps extracts two distinct representations from a single forward pass of the LLM:

  1. Dense Representation: The hidden state of the final token in the prompt is used as a high-dimensional semantic vector.
  2. Sparse Representation: The logits (probability distribution) for the next predicted token are extracted. This distribution is sparsified by retaining weights only for tokens that appear in the original document, creating a weighted bag-of-words.
Figure 1: The PromptReps architecture extracts dense vectors from hidden states and sparse representations from next-token logits to populate dual indices.

These representations are stored in an Approximate Nearest Neighbor (ANN) index and a traditional inverted index, respectively. Retrieval is performed by linearly interpolating the scores from both indices.

Experimental Evaluation

The authors evaluated PromptReps on major benchmarks including MSMARCO, TREC Deep Learning (2019 & 2020), and the BEIR suite. The method was tested across various LLM architectures, including Llama 3 (8B and 70B), Mistral-7B, and Phi-3-mini.

Results and Discussion

The experiments demonstrate that the hybrid combination of dense and sparse representations significantly outperforms either method in isolation.

Figure 2: Comparison of MRR@10 across different LLMs, showing that hybrid retrieval consistently outperforms dense-only or sparse-only approaches.

Key Findings

  • The Hybrid Advantage: While dense embeddings from LLMs often struggle in zero-shot settings, the sparse component provides critical keyword robustness. On the BEIR benchmark, PromptReps (Llama 3-70B) achieved an average nDCG@10 of 45.97, surpassing the BM25 baseline (43.70).
  • Scaling Effects: Retrieval performance improves as the underlying LLM scales. Moving from Llama 3-8B to 70B resulted in substantial gains, indicating that stronger generative capabilities translate to better document representations.
  • Zero-Shot Efficiency: PromptReps achieves results competitive with state-of-the-art supervised models like E5-mistral without requiring the massive GPU resources typically needed for contrastive training.

Conclusion

PromptReps demonstrates that generative LLMs can be effectively repurposed as first-stage retrievers through clever prompting. By extracting both dense and sparse signals, the method achieves high performance in zero-shot scenarios. While query latency remains a challenge compared to smaller encoders, the elimination of the training phase and the ability to leverage scaling laws make PromptReps a compelling new direction for information retrieval.

Summary updated

BibTeX
@inproceedings{zhuang2024promptreps,
  author = {Shengyao Zhuang and Xueguang Ma and Bevan Koopman and Jimmy Lin and Guido Zuccon},
  title = {{PromptReps: Prompting Large Language Models to Generate Dense and Sparse Representations for Zero-Shot Document Retrieval}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.250},
}
Figure