TL;DR
This paper investigates how to use Large Language Models (LLMs) to generate high-quality synthetic training data for e-commerce semantic parsers. By conditioning prompts on "personas"—including personality traits, syntactic styles, and demographics—the authors show that while diversity improves parser performance, it often comes at the cost of "hallucinated" queries that drift from the original search intent.
What Problem Does This Paper Address?
Building natural-language interfaces for e-commerce (like "find me a blue SUV under £20k with low mileage") requires semantic parsers that map queries to structured API fields. Training these parsers usually requires massive amounts of labeled data, which is expensive to collect for new domains (the "cold-start" problem).
LLMs can generate synthetic (query, intent) pairs to fill this gap. However, simple prompts often produce repetitive, "templated" text. While researchers use "persona prompting" to inject diversity, aggressive prompting can lead to hallucinations: the LLM might drop constraints (forgetting "under £20k"), add unsupported ones, or reverse logic (changing "minimum price" to "maximum price"). This paper explores the fundamental tension between useful diversity and hallucinated variety.
Methods
The authors generated 118,098 synthetic pairs using six state-of-the-art reasoning LLMs (including GPT-5.4 and Claude Opus 4.6). They tested a $2^3$ grid of prompt "bundles" across three axes, plus a standalone demographic axis:
- IPIP Big Five (p): Modulating linguistic cues based on personality (e.g., "Conscientiousness" leads to explicit constraint listing).
- Attribute-style Adjectives (a): Injecting subjective words like "reliable" or "sleek" that aren't in the formal schema.
- Syntax Modes (s): Forcing styles like noun phrases, telegraphic fragments, or natural questions.
- UK Demographics (u): Using real UK census data to create personas from specific postcodes.
They evaluated the resulting data through three lenses:
- Internal Diversity: Measured via Self-BLEU-4 and Distinct-2.
- Distributional Alignment: Using a novel metric (vMF sKL) to see how closely synthetic data matches real human queries in embedding space.
- Downstream Utility: Training a Flan-T5 parser on the synthetic data and testing it on real human-annotated queries.
Datasets
- Synthetic Corpus: 118,098 pairs across 9 prompt configurations.
- Human Reference: 1,214 pairs collected via two protocols (one where users wrote queries for specific "scenario cards" and another where they used a live website).
- Production Reference: 15,304 anonymized queries from live UK automotive search traffic.
- Schema: A high-dimensional 47-field UK automotive schema (make, model, mileage, finance terms, etc.).
Key Insights and Ideas
- The "Diversity Engine" is Syntax: Syntactic conditioning (s) is the primary driver of lexical diversity and is the only axis that moves synthetic generations toward the "short and messy" style of real production traffic.
- The "Function-Word Tax": When LLMs are told to write in "telegraphic" or "noun phrase" styles, they often drop critical "operator words" like under, within, or from. This leads to the highest hallucination rates (38–42%) because the parser can no longer tell if a number refers to a minimum or maximum bound.
- Personality is the "Cleanest" Gain: The IPIP personality bundle (p) provides the most reliable performance boost (+1.1 to +2.1 F1 points) with the lowest hallucination rate (28%). It acts like a "paraphrasing dial" that widens wording while keeping the underlying logical structure intact.
- Alignment as a Proxy: The authors found that Alignment sKL (the divergence between synthetic and human query distributions) is a very strong unlabeled proxy for parser performance ($\rho = -0.917$). This means you can estimate how good your synthetic data is without needing expensive labels, provided you have a small sample of representative human queries.
Key Results
- Every conditioned parser beat the vanilla prompt baseline. The best-performing configuration was the full
pas(personality + adjectives + syntax) bundle. - Parser Utility: The
pasconfiguration achieved an F1 of 79.62, compared to 75.98 for the vanilla baseline (see Figure 2). - Hallucination Audit: An LLM-as-judge committee found that 37% of "universal failures" (queries no parser could solve) were actually due to the generator hallucinating or dropping constraints from the intent.
Limitations and Future Work
The study is limited to a single domain (UK automotive) and a specific schema. The authors also note a "UK-cultural exposure confound": models trained primarily on US data might struggle with the nuances of UK demographic personas or automotive terminology.
Future work could explore "hallucination-aware" filtering to automatically prune unfaithful synthetic pairs before training, or investigate how smaller, non-reasoning models handle these complex persona instructions. The authors recommend the IPIP prompt as the "safe default" for practitioners looking to bootstrap new search domains.
Summary updated