- A Synthetic Query Generation Approach for Improving Zero-Shot Relevance Prediction
Introduction
In the field of Information Retrieval (IR), the "cold start" problem remains a significant challenge. Building high-performing search or relevance models for new domains—such as niche scientific fields or specialized product catalogs—is difficult when labeled training data is non-existent. The current state-of-the-art approach involves using Large Language Models (LLMs) to generate synthetic query-document pairs (QGen) to train downstream models. However, standard QGen methods often struggle to produce high-quality negative examples, which are crucial for training robust rankers.
In the paper "It’s All Relative!", researchers from Google Research propose that the limitation lies in how we prompt LLMs. Rather than asking a model to generate a relevant query in isolation, they argue that LLMs are far more effective when asked to generate relevant and irrelevant queries relative to each other. This comparative reasoning leads to the development of the GENERATE-PAIRWISE strategy.
Methodology: The Power of Pairwise Prompting
The core innovation of this work is the transition from absolute to relative query generation. Existing pipelines typically generate a relevant query and then either use a separate retriever to find "hard negatives" or prompt the LLM to generate an irrelevant query from scratch. The latter often results in "easy negatives"—queries that are so unrelated to the document that they provide little signal for training.
The GENERATE-PAIRWISE Pipeline
- Pairwise Prompting: The method utilizes a few-shot prompting strategy (using only 10 examples) that presents the LLM with a document followed by a pair: a
relevant_queryand anirrelevant_query. - Relative Conditioning: By forcing the model to generate the irrelevant query immediately after the relevant one, the model anchors the negative example to the same context. This naturally produces "hard negatives"—queries that are thematically related to the document but fail to meet specific relevance criteria.
- Round-Trip Filtration: To ensure the integrity of the synthetic data, the LLM is used as a self-filter. It predicts the relevance label for its own generated queries; if the prediction contradicts the intended label, the query is discarded.
- Downstream Training: A smaller, efficient model (such as mT5-XXL) is then trained on this high-quality synthetic data to perform zero-shot prediction in the target domain.
Experimental Evaluation
The researchers evaluated the approach across two major benchmarks:
- BEIR Benchmark: Seven diverse datasets including TREC-COVID (bio-medical), FiQA-2018 (finance), Touché20 (argument retrieval), and DBPedia (entity retrieval).
- E-commerce/Shopping: The WANDS and ESCI datasets, which require distinguishing between nuanced labels like "Exact" match versus "Substitute."
Results and Discussion
The results demonstrate that comparative reasoning significantly enhances the quality of synthetic training data. The GENERATE-PAIRWISE approach achieved an average improvement of +8.6 NDCG@10 over standard synthetic generation methods across multiple benchmarks.
Key findings from the evaluation include:
- Surpassing Human-Labeled Transfer Learning: On datasets like TREC-COVID, Touché, and Climate-FEVER, the model trained solely on synthetic pairwise data outperformed a "Skyline" model trained on 6 million real human-labeled pairs from MS-MARCO.
- Fine-Grained Relevance: In e-commerce tasks (WANDS), the pairwise method achieved an NDCG@10 of 0.8882, significantly higher than previous fine-tuned models (0.8553).
- Efficiency in Prompting: Despite using only 10 examples to prime the LLM, the resulting data was more effective than massive out-of-domain datasets, highlighting that quality and task structure are more important than raw data volume.
Conclusion
This research provides a compelling case for restructuring how we use LLMs for data augmentation. By framing query generation as a comparative task rather than an absolute one, we can generate higher-quality hard negatives that significantly improve zero-shot performance. The "It's All Relative" approach simplifies the synthetic data pipeline by building hard-negative generation directly into the prompting process, setting a new standard for zero-shot relevance prediction.
Summary updated