Syntriever: How to Train Your Retriever with Synthetic Data from LLMs

Minsang Kim, Seung Jun Baek

Abstract

Large Language Models (LLMs) have boosted progress in many AI applications. Recently, there were attempts to distill the vast knowledge of LLMs into information retrieval systems. Those distillation methods mostly use output probabilities of LLMs which are unavailable in the latest black-box LLMs. We propose Syntriever, a training framework for retrievers using synthetic data from black-box LLMs. Syntriever consists of two stages. Firstly in the distillation stage, we synthesize relevant and plausibly irrelevant passages and augmented queries using chain-of-thoughts for the given queries. LLM is asked to self-verify the synthetic data for possible hallucinations, after which retrievers are trained with a loss designed to cluster the embeddings of relevant passages. Secondly in the alignment stage, we align the retriever with the preferences of LLMs. We propose a preference modeling called partial Plackett-Luce ranking to learn LLM preferences with regularization which prevents the model from deviating excessively from that trained in the distillation stage. Experiments show that Syntriever achieves state-of-the-art performances on benchmark datasets from various domains in nDCG@K.

Introduction

In Retrieval-Augmented Generation (RAG) systems, the retriever is the critical component responsible for identifying relevant documents to ground the LLM's responses. While distilling LLM expertise into smaller, efficient retrieval models is a common goal, traditional methods often rely on internal probability distributions (log-probs) of the teacher model. However, the most advanced modern LLMs, such as GPT-4o, are "black boxes" that only provide text outputs. Syntriever addresses this challenge by providing a framework to effectively distill an LLM's knowledge into a retriever using only its text generations.

The Syntriever Framework

Syntriever operates in two distinct stages, mirroring the Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) pipeline used to train the LLMs themselves.

The two-stage Syntriever framework: Stage 1 focuses on distilling LLM knowledge through synthetic data generation, while Stage 2 aligns the retriever with LLM preferences.

Stage 1: Distillation through Synthesis

Instead of simple query-document pairs, Syntriever leverages the LLM to create a rich synthetic dataset through several specialized processes:

  • Query Decomposition: Using Chain-of-Thought (CoT) prompting ($P_{cot}$), the framework breaks complex queries into simpler sub-queries. This helps the retriever understand the multi-step "plan" required to find a comprehensive answer.
  • Hard Negative Synthesis: The LLM is prompted ($P_-$) to generate passages that are "plausible but irrelevant," providing high-quality negative samples that challenge the retriever to distinguish between topical similarity and actual relevance.
  • Hallucination-Aware Relabeling: A key innovation is the treatment of hallucinations. If the LLM generates a "positive" passage ($P_+$) that it later identifies as a hallucination during a self-verification step ($P_{Relabel}$), the framework relabels it as a "hard negative." This teaches the retriever to avoid the exact kind of misinformation that might otherwise trick a dense model.
  • Clustering Loss: The model is trained using a modified Soft Nearest-Neighbor (SNN) loss, which clusters the embeddings of the original query, its sub-queries, and the generated positive passages together in the embedding space.

Stage 2: Alignment from LLM Feedback

After initial distillation, the retriever is "aligned" with the LLM's fine-grained preferences. The LLM acts as a judge ($P_{compare}$), comparing pairs of passages retrieved by the Stage 1 model and selecting the more relevant one ($y_w \succ y_l$).

To learn from these preferences, the authors propose Partial Plackett-Luce (PL) Ranking. Unlike standard preference models that only consider the compared pair, the Partial PL loss incorporates "in-batch negatives" (irrelevant samples from other queries). This acts as a form of regularization, preventing the model from "forgetting" how to distinguish relevant from irrelevant documents while it learns the subtle nuances of ranking.

Experimental Results

Syntriever was evaluated on the BeIR (Benchmarking Information Retrieval) suite, covering diverse domains including search engines (MSMARCO), fact-checking (FEVER), and specialized fields like medicine (NFCorpus) and finance (FiQA).

Comparative performance of Syntriever on the BeIR benchmark, showing significant improvements over existing dense retrieval models.

As shown in the results, Syntriever demonstrates significant performance gains:

  • State-of-the-Art Performance: Syntriever outperformed established models like BGE-M3-EN, E5, and Nomic-Embed on nearly every BeIR task. On the MSMARCO dataset, it achieved an nDCG@10 of 50.1, a substantial improvement over its base model.
  • Zero-Shot Generalization: The framework proved highly effective in zero-shot scenarios. When trained on MSMARCO and tested on unseen domains, Syntriever maintained a significant lead, suggesting that LLM-generated data provides better generalization than human-labeled data alone.
  • Teacher Efficiency: Interestingly, the research found that GPT-4o-mini is often as effective a teacher as the full GPT-4o for distillation, indicating that high-quality retrieval training can be achieved with lower compute costs.

Conclusion

Syntriever represents a significant step forward in retriever training by successfully bridging the gap between black-box LLMs and dense retrieval systems. By treating hallucinations as valuable training data and introducing a novel partial ranking loss for alignment, the framework achieves state-of-the-art results across a wide array of information retrieval tasks. Future work will explore scaling this pipeline with smaller open-source teacher models and more complex ranking structures.

Summary updated

Abstract

LLMs have boosted progress in many AI applications. Recently, there were attempts to distill the vast knowledge of LLMs into information retrieval systems. Those distillation methods mostly use output probabilities of LLMs which are unavailable in the latest black-box LLMs. We propose Syntriever, a training framework for retrievers using synthetic data from black-box LLMs. Syntriever consists of two stages. Firstly in the distillation stage, we synthesize relevant and plausibly irrelevant passages and augmented queries using chain-of-thoughts for the given queries. LLM is asked to self-verify the synthetic data for possible hallucinations, after which retrievers are trained with a loss designed to cluster the embeddings of relevant passages. Secondly in the alignment stage, we align the retriever with the preferences of LLMs. We propose a preference modeling called partial Plackett-Luce ranking to learn LLM preferences with regularization which prevents the model from deviating excessively from that trained in the distillation stage. Experiments show that Syntriever achieves state-of-the-art performances on benchmark datasets from various domains in nDCG@K. the source code is available in https://github.com/kmswin1/Syntriever
BibTeX
@inproceedings{kim2025syntriever,
  author = {Minsang Kim and Seung Jun Baek},
  title = {{Syntriever: How to Train Your Retriever with Synthetic Data from LLMs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  url = {https://openreview.net/forum?id=l6AeKHFFnG},
}
Figure