Abstract
Llama2Vec is a novel unsupervised adaptation method that transforms Large Language Models (LLMs) into high-performance encoders for dense retrieval. By employing two specialized pretext tasks—Embedding-Based Auto-Encoding (EBAE) and Embedding-Based Auto-Regression (EBAR)—it shifts the LLM's focus from local next-token prediction to global semantic representation. Llama2Vec sets new state-of-the-art benchmarks on MS MARCO and BEIR, demonstrating the latent potential of LLMs as retrieval backbones without requiring massive labeled datasets.
Introduction
In Information Retrieval (IR), "dense retrieval" relies on representing queries and documents as single vectors (embeddings) in a shared space. While LLMs like Llama-2 possess incredible semantic understanding, they are fundamentally auto-regressive models trained for next-token prediction. This creates a "local vs. global" discrepancy: an LLM’s internal representations are optimized for local context rather than summarizing the entire meaning of a text block into a single discriminative vector. Llama2Vec addresses this challenge by adapting these generative giants into effective backbone encoders for retrieval.
Methodology
The authors introduce Llama2Vec, an unsupervised "continuation of pre-training" consisting of two pretext tasks designed to force the model to generate global embeddings:
- EBAE (Embedding-Based Auto-Encoding): The model is prompted to reconstruct the entire input sentence based solely on a single text embedding generated at the end of the sequence. This encourages induction—capturing the global semantics of the text itself.
- EBAR (Embedding-Based Auto-Regression): The model is prompted to predict the entire next sentence based on the current sentence's embedding. This encourages deduction—capturing the relationship between correlated pieces of text, which is essential for tasks like Question Answering.
To optimize training, the authors developed a "joint prompt" scheme: [Input] SELF </s> NEXT </s>. By using a custom attention mask, they ensure the "SELF" and "NEXT" tokens are mutually invisible during training, reducing computation costs by nearly 50% compared to processing the text twice.
Evaluation and Results
The model was adapted using the Wikipedia corpus and evaluated across several standard benchmarks. For supervised evaluation, the authors used MS MARCO (Passage and Document retrieval). For zero-shot generalization, they utilized the BEIR benchmark and the Llama Index.
Key Results
Llama2Vec achieved impressive performance across the board, often outperforming models much larger or those trained on more data:
- MS MARCO Passage: Achieved an MRR@10 of 43.1, setting a new state-of-the-art for its class and significantly outperforming the baseline RepLLaMA (41.2).
- BEIR (Zero-Shot): Reached an average NDCG@10 of 56.4, surpassing established models like BGE-M3 and OpenAI’s
text-embedding-3-large. - Dominating BM25: Llama2Vec outperformed the classic BM25 algorithm on 13 out of 14 BEIR datasets, showing a 31% relative improvement in average performance.
Insights and Discussion
The core innovation of Llama2Vec is "unlocking" an LLM's latent potential for retrieval by forcing it to predict a whole vocabulary distribution from a single vector. This process significantly increases the lexical similarity between query and document embeddings even before supervised fine-tuning. Furthermore, the use of specific prompts (SELF and NEXT) allows the model to be "steered" during inference; for instance, using the NEXT prompt for a query and the SELF prompt for a document naturally aligns with the Question-Answer relationship.
Limitations and Future Work
While effective, the current study focused on the 7B parameter version of Llama-2; the impact on larger models (e.g., 70B) remains to be explored. Additionally, the work is currently English-centric. Future research will focus on extending this unsupervised adaptation to multilingual scenarios and finding ways to maintain this performance in smaller, distilled models to reduce inference costs in production environments.
Summary updated