TL;DR
Synthetic query generation (QGen) has emerged as a powerful tool for training search engines, but how does it handle the messy, multi-graded reality of e-commerce? This paper reveals that standard QGen methods fail in nuanced domains because they treat relevance as a simple binary (yes/no) problem. By introducing "label-conditioned" QGen, the authors show that while teaching language models to respect different grades of relevance helps, current generative models still fundamentally struggle to create faithful, fine-grained synthetic data.
What Problem Does This Paper Address?
Building high-quality search and recommendation systems for specialized domains like e-commerce is notoriously difficult due to a lack of data. User logs are proprietary, privacy regulations are strict, and human annotation is incredibly expensive.
To circumvent this, researchers often use Large Language Models (LLMs) for Query Generation (QGen)—feeding a product description into a model and asking it to generate a plausible search query. While this works well for binary relevance (e.g., "Is this document relevant to the query or not?"), real-world e-commerce requires much more nuance. A search for a "blue sweater" might return an exact match (a blue sweater), a substitute (a red sweater), a complement (a matching scarf), or something completely irrelevant.
Standard QGen approaches collapse this multi-graded relevance space into a simple binary choice, completely losing the nuance required to train high-performing product rankers.
Methods
The authors evaluate two primary families of QGen: Finetune-Based (using mT5-XXL) and Prompt-Based (using FLAN-137B). They contrast traditional "Vanilla" QGen against their proposed adaptation: Label-Conditioned QGen (LabelCond QGen).
- Vanilla QGen: The model is trained purely on highly relevant query-product pairs to generate a single relevant query given a product description.
- LabelCond QGen (Proposed): Instead of hiding the relevance grades, the authors prepend the specific relevance label directly to the input context during training or prompting. This forces the model to learn the distinct linguistic characteristics of queries across different relevance tiers (e.g., generating a broader query for a Substitute vs. a highly specific one for an Exact match).
Datasets
The study leverages four distinct datasets spanning general web search and e-commerce:
- MS-MARCO: A massive, general-purpose web search dataset used here as an out-of-domain baseline (binary relevance).
- ESCI (Amazon Shopping Queries): A large e-commerce dataset containing 2.6 million query-product pairs rated across four classes: Exact (E), Substitute (S), Complement (C), and Irrelevant (I). Used for in-domain training.
- WANDS & HomeDepot: Two independent e-commerce benchmarks focusing on home improvement, featuring multi-class human relevance judgments. These serve strictly as the target evaluation sets to test zero-shot performance.
Key Insights and Ideas
- The Nuance Collapse: Contrary to recent literature claiming QGen outperforms traditional cross-domain transfer learning, this paper shows the opposite holds true for fine-grained tasks. Vanilla QGen fails because it cannot distinguish between different shades of relevance.
- Label Prepending is Not a Silver Bullet: Simply adding a text token like
Label: Substituteto the prompt isn't enough to guide the LLM. The authors discovered that the finetuned ESCI model produced identical, duplicate queries across different relevance labels for 81% of the products. The model frequently ignored the condition and defaulted to generating a generic, highly relevant query. - Domain Alignment Trumps Dataset Size: Models trained or augmented using the smaller, e-commerce-specific ESCI dataset consistently beat models trained on the massive, general-purpose MS-MARCO dataset. Knowing the domain language matters more than sheer data volume.
Key Results
The downstream utility of the generated data was evaluated using Normalized Discounted Cumulative Gain (NDCG), where higher scores indicate better ranking capabilities.
- Transfer Learning Wins: On the WANDS dataset, simple zero-shot transfer learning from the ESCI dataset achieved the highest performance (0.8927 NDCG@10), outperforming all synthetic data approaches.
- Label Conditioning Helps: Within the QGen space, conditioning on labels provided a massive boost. For instance, on WANDS,
Finetune-Based-LabelCond (MS-MARCO)outperformed its vanilla counterpart by +3.3 NDCG@10 points. - Prompt-Based Models Struggle with Nuance: Few-shot prompting (
Prompt-Based-LabelCond) severely underperformed compared to finetuned models, trailing the best finetuned QGen model by over 20 NDCG points on WANDS, indicating that current LLMs struggle to grasp multi-class relevance relationships purely from a few examples.
Limitations and Future Work
The primary limitation identified is the lack of faithfulness in the generated queries; the models struggle to internalize the ordered relationship of a multi-class label space (e.g., understanding that a Substitute is closer to Exact than an Irrelevant product).
Moving forward, the authors suggest that future QGen architectures must explicitly enforce label constraints throughout the loss function rather than relying on simple text prefixing. Additionally, they highlight the need for better intrinsic evaluation metrics so researchers can evaluate synthetic data quality without needing to train an entire downstream model every time.
Summary updated