TL;DR
This paper introduces Syntriever, a novel two-stage framework for training information retrieval (IR) models using synthetic data generated by large language models (LLMs). Syntriever effectively distills LLM knowledge into smaller, more efficient retrievers, achieving state-of-the-art performance on various benchmarks without requiring direct access to LLM output probabilities.
What Problem Does This Paper Address?
Large Language Models (LLMs) have revolutionized many AI applications, but their knowledge is often siloed and inaccessible for real-time information retrieval. Retrieval-Augmented Generation (RAG) systems aim to bridge this gap by incorporating external knowledge, but training effective retrievers for RAG has been challenging, especially with the rise of black-box LLMs where direct probability outputs are unavailable. Existing methods often rely on these probabilities for knowledge distillation. This paper addresses the challenge of distilling LLM knowledge into efficient retrievers when only synthetic text generation is possible, aiming to create powerful retrievers that can leverage the vast knowledge embedded within LLMs.
Methods
Syntriever employs a two-stage approach:
- Stage 1: Distillation Stage: This stage focuses on synthesizing data to train the retriever.
- Chain-of-Thought (CoT) for Augmented Queries: LLMs are prompted to break down complex queries into simpler sub-queries, creating an "augmented query" that provides richer context.
- Synthetic Positive and Hard-Negative Passages: LLMs generate plausible positive passages (relevant to the query) and plausible but irrelevant "hard-negative" passages.
- Self-Verification and Relabeling: LLMs are used to verify synthetic positive passages for hallucinations. If a passage is deemed a hallucination, it's relabeled as a hard-negative. This leverages the LLM's self-verification capability and turns potential errors into useful training signals.
- Modified Soft Nearest-Neighbor (SNN) Loss: A custom loss function is used to cluster embeddings of relevant passages (original, CoT-augmented, and synthetic positives) in the embedding space, while repelling irrelevant passages.
- Stage 2: Alignment Stage: This stage aligns the retriever with LLM preferences.
- LLM Preference Feedback: The retriever, trained in Stage 1, retrieves top-K passages for a given query. Pairs of these passages are then presented to an LLM, which indicates its preference.
- Partial Plackett-Luce Ranking: A novel preference modeling technique is introduced. It models the LLM's preference for a positive passage over a negative passage, while also considering the in-batch negative passages as less preferred. This method acts as a regularizer, preventing the retriever from deviating too much from the knowledge learned during the distillation stage.
Datasets
The experiments were conducted on several benchmark datasets from the BeIR (Benchmark for Information Retrieval) collection, which is designed for evaluating retrieval models across various domains. Specific datasets used include:
- Supervised Fine-Tuning: MSMARCO, HotpotQA, FIQA, Fever, SciFact, NFCorpus, NQ.
- Zero-shot Transfer: The same BeIR datasets were used, but models were first fine-tuned on large datasets like MSMARCO and NQ.
The paper also mentions the use of MSMARCO and NQ for pre-training in the zero-shot transfer setting.
Key Insights and Ideas
- Synthetic Data as a Powerful Distillation Tool: The core idea that LLMs can generate high-quality synthetic data (queries, passages) to effectively train smaller, specialized retrieval models is a significant contribution. This bypasses the need for direct probability outputs, making it applicable to black-box LLMs.
- Hallucinations as Hard Negatives: Instead of discarding hallucinated passages, Syntriever cleverly relabels them as hard negatives. This turns a potential weakness of LLM generation into a strength for training robust retrievers.
- Two-Stage Training Mimics LLM Training: The distillation (SFT-like) and alignment (RLHF-like) stages mirror the successful training paradigm of LLMs, suggesting a generalizable approach for knowledge transfer.
- Partial Plackett-Luce for Alignment: This novel preference modeling approach is crucial for aligning retrievers with LLM preferences while maintaining the learned embeddings from the distillation stage, acting as a form of regularization.
- Smaller LLMs Can Be Effective Teachers: The study shows that even "weaker" but cheaper LLMs (like GPT-40-mini) can be effective teachers, sometimes even outperforming larger models in distillation, suggesting a more accessible path for training retrievers.
Key Results
- State-of-the-Art Performance: Syntriever achieved state-of-the-art results on seven BeIR benchmark datasets in supervised fine-tuning, with improvements up to 18.6% in nDCG@10.
- Strong Zero-Shot Transfer: In zero-shot transfer settings, Syntriever achieved the best performance on 8 out of 15 datasets and the second best on one.
- Significant Gains over Base Models: Syntriever consistently improved the performance of various base encoders (e.g., E5, ColBERT, SBERT), demonstrating its framework's broad applicability.
- Effectiveness of Components: Ablation studies confirmed that both the synthesized queries/passages (distillation stage) and the LLM preference alignment (alignment stage) significantly contribute to performance gains.
- Quality of Synthetic Positives: Interestingly, GPT-40 indicated that 88% of synthetic positive passages (after self-verification) contained direct contexts to the true answer, compared to 84% for ground truth passages, suggesting high-quality synthetic data generation.
Limitations and Future Work
- Computational Cost: Syntriever requires LLM inferences for generating synthetic data and preference feedback, which incurs additional computational costs compared to methods that only fine-tune text encoders.
- Dependency on LLM Quality: The performance of Syntriever is inherently tied to the quality of the LLM used for data generation and preference feedback.
- Future Work: The authors suggest that Syntriever is widely applicable to retrieval systems and that distillation/alignment through smaller LLMs could become an increasingly good alternative, especially under fixed compute budgets. They also highlight the potential for combining Syntriever with diverse base retrievers and LLMs.
Summary updated