Exploring the Viability of Synthetic Query Generation for Relevance Prediction

Aditi Chaudhary, Karthik Raman, Krishna Srinivasan, Kazuma Hashimoto, Mike Bendersky, Marc Najork

TL;DR

Synthetic query generation (QGen) has emerged as a powerful tool for training search engines, but how does it handle the messy, multi-graded reality of e-commerce? This paper reveals that standard QGen methods fail in nuanced domains because they treat relevance as a simple binary (yes/no) problem. By introducing "label-conditioned" QGen, the authors show that while teaching language models to respect different grades of relevance helps, current generative models still fundamentally struggle to create faithful, fine-grained synthetic data.

What Problem Does This Paper Address?

Building high-quality search and recommendation systems for specialized domains like e-commerce is notoriously difficult due to a lack of data. User logs are proprietary, privacy regulations are strict, and human annotation is incredibly expensive.

To circumvent this, researchers often use Large Language Models (LLMs) for Query Generation (QGen)—feeding a product description into a model and asking it to generate a plausible search query. While this works well for binary relevance (e.g., "Is this document relevant to the query or not?"), real-world e-commerce requires much more nuance. A search for a "blue sweater" might return an exact match (a blue sweater), a substitute (a red sweater), a complement (a matching scarf), or something completely irrelevant.

Standard QGen approaches collapse this multi-graded relevance space into a simple binary choice, completely losing the nuance required to train high-performing product rankers.

Methods

The authors evaluate two primary families of QGen: Finetune-Based (using mT5-XXL) and Prompt-Based (using FLAN-137B). They contrast traditional "Vanilla" QGen against their proposed adaptation: Label-Conditioned QGen (LabelCond QGen).

Figure 1: System overview of different approaches we examine for improving relevance prediction on WANDS and HomeDepot. zero-shot application is a popular transfer learning strategy where labeled data from existing datasets (ESCI (in-domain) or MS- MARCO (out-of-domain) can be used directly to train downstream task model. Generate Queries approach uses labeled data to train Query Generation (QGen) models – vanilla QGen which most existing works use, where document information, in our case shopping-product information, is used to generate a relevant query and LabelCond QGen which we introduce to generate queries by additionally conditioning on all relevance labels. The generated queries are then used to finetune the downstream model.
  • Vanilla QGen: The model is trained purely on highly relevant query-product pairs to generate a single relevant query given a product description.
  • LabelCond QGen (Proposed): Instead of hiding the relevance grades, the authors prepend the specific relevance label directly to the input context during training or prompting. This forces the model to learn the distinct linguistic characteristics of queries across different relevance tiers (e.g., generating a broader query for a Substitute vs. a highly specific one for an Exact match).
QGen input and output formats

Datasets

The study leverages four distinct datasets spanning general web search and e-commerce:

  • MS-MARCO: A massive, general-purpose web search dataset used here as an out-of-domain baseline (binary relevance).
  • ESCI (Amazon Shopping Queries): A large e-commerce dataset containing 2.6 million query-product pairs rated across four classes: Exact (E), Substitute (S), Complement (C), and Irrelevant (I). Used for in-domain training.
  • WANDS & HomeDepot: Two independent e-commerce benchmarks focusing on home improvement, featuring multi-class human relevance judgments. These serve strictly as the target evaluation sets to test zero-shot performance.

Key Insights and Ideas

  • The Nuance Collapse: Contrary to recent literature claiming QGen outperforms traditional cross-domain transfer learning, this paper shows the opposite holds true for fine-grained tasks. Vanilla QGen fails because it cannot distinguish between different shades of relevance.
  • Label Prepending is Not a Silver Bullet: Simply adding a text token like Label: Substitute to the prompt isn't enough to guide the LLM. The authors discovered that the finetuned ESCI model produced identical, duplicate queries across different relevance labels for 81% of the products. The model frequently ignored the condition and defaulted to generating a generic, highly relevant query.
  • Domain Alignment Trumps Dataset Size: Models trained or augmented using the smaller, e-commerce-specific ESCI dataset consistently beat models trained on the massive, general-purpose MS-MARCO dataset. Knowing the domain language matters more than sheer data volume.

Key Results

The downstream utility of the generated data was evaluated using Normalized Discounted Cumulative Gain (NDCG), where higher scores indicate better ranking capabilities.

Table 2: Results on WANDS dataset where we report NDCG@5,10,20 (higher the better).
  • Transfer Learning Wins: On the WANDS dataset, simple zero-shot transfer learning from the ESCI dataset achieved the highest performance (0.8927 NDCG@10), outperforming all synthetic data approaches.
  • Label Conditioning Helps: Within the QGen space, conditioning on labels provided a massive boost. For instance, on WANDS, Finetune-Based-LabelCond (MS-MARCO) outperformed its vanilla counterpart by +3.3 NDCG@10 points.
  • Prompt-Based Models Struggle with Nuance: Few-shot prompting (Prompt-Based-LabelCond) severely underperformed compared to finetuned models, trailing the best finetuned QGen model by over 20 NDCG points on WANDS, indicating that current LLMs struggle to grasp multi-class relevance relationships purely from a few examples.

Limitations and Future Work

The primary limitation identified is the lack of faithfulness in the generated queries; the models struggle to internalize the ordered relationship of a multi-class label space (e.g., understanding that a Substitute is closer to Exact than an Irrelevant product).

Moving forward, the authors suggest that future QGen architectures must explicitly enforce label constraints throughout the loss function rather than relying on simple text prefixing. Additionally, they highlight the need for better intrinsic evaluation metrics so researchers can evaluate synthetic data quality without needing to train an entire downstream model every time.

Summary updated

Abstract

Query-document relevance prediction is a critical problem in Information Retrieval systems. This problem has increasingly been tackled using (pretrained) transformer-based models which are finetuned using large collections of labeled data. However, in specialized domains such as e-commerce and healthcare, the viability of this approach is limited by the dearth of large in-domain data. To address this paucity, recent methods leverage these powerful models to generate high-quality task and domain-specific synthetic data. Prior work has largely explored synthetic data generation or query generation (QGen) for Question-Answering (QA) and binary (yes/no) relevance prediction, where for instance, the QGen models are given a document, and trained to generate a query relevant to that document. However in many problems, we have a more fine-grained notion of relevance than a simple yes/no label. Thus, in this work, we conduct a detailed study into how QGen approaches can be leveraged for nuanced relevance prediction. We demonstrate that -- contrary to claims from prior works -- current QGen approaches fall short of the more conventional cross-domain transfer-learning approaches. Via empirical studies spanning 3 public e-commerce benchmarks, we identify new shortcomings of existing QGen approaches -- including their inability to distinguish between different grades of relevance. To address this, we introduce label-conditioned QGen models which incorporates knowledge about the different relevance. While our experiments demonstrate that these modifications help improve performance of QGen techniques, we also find that QGen approaches struggle to capture the full nuance of the relevance label space and as a result the generated queries are not faithful to the desired relevance label.
BibTeX
@inproceedings{chaudhary2023exploring,
  author = {Aditi Chaudhary and Karthik Raman and Krishna Srinivasan and Kazuma Hashimoto and Mike Bendersky and Marc Najork},
  title = {{Exploring the Viability of Synthetic Query Generation for Relevance Prediction}},
  booktitle = {eCom @ SIGIR},
  year = {2023},
}
Figure