Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search

Filippos Ventirozos, Tom Armitage, Mahmoud Oshagh, Peter A. Appleby, Matthew ShardlowarXiv 2026 · arXiv

TL;DR

This paper investigates how to use Large Language Models (LLMs) to generate high-quality synthetic training data for e-commerce semantic parsers. By conditioning prompts on "personas"—including personality traits, syntactic styles, and demographics—the authors show that while diversity improves parser performance, it often comes at the cost of "hallucinated" queries that drift from the original search intent.

What Problem Does This Paper Address?

Building natural-language interfaces for e-commerce (like "find me a blue SUV under £20k with low mileage") requires semantic parsers that map queries to structured API fields. Training these parsers usually requires massive amounts of labeled data, which is expensive to collect for new domains (the "cold-start" problem).

LLMs can generate synthetic (query, intent) pairs to fill this gap. However, simple prompts often produce repetitive, "templated" text. While researchers use "persona prompting" to inject diversity, aggressive prompting can lead to hallucinations: the LLM might drop constraints (forgetting "under £20k"), add unsupported ones, or reverse logic (changing "minimum price" to "maximum price"). This paper explores the fundamental tension between useful diversity and hallucinated variety.

Methods

The authors generated 118,098 synthetic pairs using six state-of-the-art reasoning LLMs (including GPT-5.4 and Claude Opus 4.6). They tested a $2^3$ grid of prompt "bundles" across three axes, plus a standalone demographic axis:

  1. IPIP Big Five (p): Modulating linguistic cues based on personality (e.g., "Conscientiousness" leads to explicit constraint listing).
  2. Attribute-style Adjectives (a): Injecting subjective words like "reliable" or "sleek" that aren't in the formal schema.
  3. Syntax Modes (s): Forcing styles like noun phrases, telegraphic fragments, or natural questions.
  4. UK Demographics (u): Using real UK census data to create personas from specific postcodes.

They evaluated the resulting data through three lenses:

  • Internal Diversity: Measured via Self-BLEU-4 and Distinct-2.
  • Distributional Alignment: Using a novel metric (vMF sKL) to see how closely synthetic data matches real human queries in embedding space.
  • Downstream Utility: Training a Flan-T5 parser on the synthetic data and testing it on real human-annotated queries.

Datasets

  • Synthetic Corpus: 118,098 pairs across 9 prompt configurations.
  • Human Reference: 1,214 pairs collected via two protocols (one where users wrote queries for specific "scenario cards" and another where they used a live website).
  • Production Reference: 15,304 anonymized queries from live UK automotive search traffic.
  • Schema: A high-dimensional 47-field UK automotive schema (make, model, mileage, finance terms, etc.).

Key Insights and Ideas

  • The "Diversity Engine" is Syntax: Syntactic conditioning (s) is the primary driver of lexical diversity and is the only axis that moves synthetic generations toward the "short and messy" style of real production traffic.
  • The "Function-Word Tax": When LLMs are told to write in "telegraphic" or "noun phrase" styles, they often drop critical "operator words" like under, within, or from. This leads to the highest hallucination rates (38–42%) because the parser can no longer tell if a number refers to a minimum or maximum bound.
  • Personality is the "Cleanest" Gain: The IPIP personality bundle (p) provides the most reliable performance boost (+1.1 to +2.1 F1 points) with the lowest hallucination rate (28%). It acts like a "paraphrasing dial" that widens wording while keeping the underlying logical structure intact.
  • Alignment as a Proxy: The authors found that Alignment sKL (the divergence between synthetic and human query distributions) is a very strong unlabeled proxy for parser performance ($\rho = -0.917$). This means you can estimate how good your synthetic data is without needing expensive labels, provided you have a small sample of representative human queries.
Figure 1: Internal diversity per prompt configuration, ordered by Self-BLEU-4 (more diverse on the left). Coloured bars are the mean across 6 generators; the light-grey shadow behind each bar spans ±1 std across LLMs. Tall shadows flag configurations whose diver- sity depends heavily on the generator; short shadows indicate that the prompt itself, not the LLM, is doing the work.

Key Results

  • Every conditioned parser beat the vanilla prompt baseline. The best-performing configuration was the full pas (personality + adjectives + syntax) bundle.
  • Parser Utility: The pas configuration achieved an F1 of 79.62, compared to 75.98 for the vanilla baseline (see Figure 2).
  • Hallucination Audit: An LLM-as-judge committee found that 37% of "universal failures" (queries no parser could solve) were actually due to the generator hallucinating or dropping constraints from the intent.
Table 2: Per-configuration parser utility. The `pas` bundle leads in F1.

Limitations and Future Work

The study is limited to a single domain (UK automotive) and a specific schema. The authors also note a "UK-cultural exposure confound": models trained primarily on US data might struggle with the nuances of UK demographic personas or automotive terminology.

Future work could explore "hallucination-aware" filtering to automatically prune unfaithful synthetic pairs before training, or investigate how smaller, non-reasoning models handle these complex persona instructions. The authors recommend the IPIP prompt as the "safe default" for practitioners looking to bootstrap new search domains.

Summary updated

Abstract

Synthetic supervision can bootstrap semantic parsers in new search domains, but prompt-induced diversity is useful only when generations are faithful to the structured intent. We explore this for cold-start e-commerce search: 118,098 synthetic pairs produced by 6 LLMs with 9 persona-conditioned prompt families on a 47-field UK automotive schema, against 1,214 human-annotated and 15,304 production queries. We show that the IPIP personality bundle improves parser F1 in all four matched contrasts (Holm-corrected $p<.001$), and all three bundles have a positive main effect over the complete $2^3$ grid. Syntax conditioning is the axis that moves generations towards real traffic. It also induces lexical diversity. Alignment sKL, the divergence between von Mises-Fisher distributions fitted to synthetic and human-annotated queries, is an unlabelled proxy for parser F1 ($ρ=-0.917$ against human queries written for simulated shopper scenarios, lower is closer). An LLM committee, checked against a blinded author pass, triages the 927 queries missed by every parser and labels 37% as hallucinated. Dropped constraints dominate. The four syntax-bearing families have the largest hallucinated shares, 38 to 42%, while IPIP alone has the lowest at 28%.
BibTeX
@misc{ventirozos2026useful,
  author = {Filippos Ventirozos and Tom Armitage and Mahmoud Oshagh and Peter A. Appleby and Matthew Shardlow},
  title = {{Useful Diversity or Hallucinated Variety? Persona-Conditioned Synthetic Query Generation for E-Commerce Search}},
  year = {2026},
  eprint = {2610.35102},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35102},
}
Figure