Making Large Language Models A Better Foundation For Dense Retrieval

Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao

Abstract

Llama2Vec is a novel unsupervised adaptation method that transforms Large Language Models (LLMs) into high-performance encoders for dense retrieval. By employing two specialized pretext tasks—Embedding-Based Auto-Encoding (EBAE) and Embedding-Based Auto-Regression (EBAR)—it shifts the LLM's focus from local next-token prediction to global semantic representation. Llama2Vec sets new state-of-the-art benchmarks on MS MARCO and BEIR, demonstrating the latent potential of LLMs as retrieval backbones without requiring massive labeled datasets.

Introduction

In Information Retrieval (IR), "dense retrieval" relies on representing queries and documents as single vectors (embeddings) in a shared space. While LLMs like Llama-2 possess incredible semantic understanding, they are fundamentally auto-regressive models trained for next-token prediction. This creates a "local vs. global" discrepancy: an LLM’s internal representations are optimized for local context rather than summarizing the entire meaning of a text block into a single discriminative vector. Llama2Vec addresses this challenge by adapting these generative giants into effective backbone encoders for retrieval.

Methodology

The authors introduce Llama2Vec, an unsupervised "continuation of pre-training" consisting of two pretext tasks designed to force the model to generate global embeddings:

  1. EBAE (Embedding-Based Auto-Encoding): The model is prompted to reconstruct the entire input sentence based solely on a single text embedding generated at the end of the sequence. This encourages induction—capturing the global semantics of the text itself.
  2. EBAR (Embedding-Based Auto-Regression): The model is prompted to predict the entire next sentence based on the current sentence's embedding. This encourages deduction—capturing the relationship between correlated pieces of text, which is essential for tasks like Question Answering.
Overview of the EBAE and EBAR pretext tasks used to adapt LLMs for retrieval.

To optimize training, the authors developed a "joint prompt" scheme: [Input] SELF </s> NEXT </s>. By using a custom attention mask, they ensure the "SELF" and "NEXT" tokens are mutually invisible during training, reducing computation costs by nearly 50% compared to processing the text twice.

Evaluation and Results

The model was adapted using the Wikipedia corpus and evaluated across several standard benchmarks. For supervised evaluation, the authors used MS MARCO (Passage and Document retrieval). For zero-shot generalization, they utilized the BEIR benchmark and the Llama Index.

Key Results

Llama2Vec achieved impressive performance across the board, often outperforming models much larger or those trained on more data:

  • MS MARCO Passage: Achieved an MRR@10 of 43.1, setting a new state-of-the-art for its class and significantly outperforming the baseline RepLLaMA (41.2).
  • BEIR (Zero-Shot): Reached an average NDCG@10 of 56.4, surpassing established models like BGE-M3 and OpenAI’s text-embedding-3-large.
  • Dominating BM25: Llama2Vec outperformed the classic BM25 algorithm on 13 out of 14 BEIR datasets, showing a 31% relative improvement in average performance.
Zero-shot retrieval performance (NDCG@10) on the BEIR benchmark across 14 diverse datasets.

Insights and Discussion

The core innovation of Llama2Vec is "unlocking" an LLM's latent potential for retrieval by forcing it to predict a whole vocabulary distribution from a single vector. This process significantly increases the lexical similarity between query and document embeddings even before supervised fine-tuning. Furthermore, the use of specific prompts (SELF and NEXT) allows the model to be "steered" during inference; for instance, using the NEXT prompt for a query and the SELF prompt for a document naturally aligns with the Question-Answer relationship.

Limitations and Future Work

While effective, the current study focused on the 7B parameter version of Llama-2; the impact on larger models (e.g., 70B) remains to be explored. Additionally, the work is currently English-centric. Future research will focus on extending this unsupervised adaptation to multilingual scenarios and finding ways to maintain this performance in smaller, distilled models to reduce inference costs in production environments.

Summary updated

BibTeX
@inproceedings{li2024making,
  author = {Chaofan Li and Zheng Liu and Shitao Xiao and Yingxia Shao},
  title = {{Making Large Language Models A Better Foundation For Dense Retrieval}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.191},
}
Figure