TL;DR
When Large Language Models (LLMs) are used to simulate user search queries for Information Retrieval (IR) evaluation, they often "cheat" by leaking information from target documents into their initial queries. This paper introduces concept provenance, a framework that maps out where query terms originate, revealing that prompt engineering cannot stop this knowledge leakage, but post-generation filtering can eliminate 99% of it.
What Problem Does This Paper Address?
In Information Retrieval (IR), researchers increasingly use LLMs to simulate human search queries to evaluate how robustly search engines perform. However, a valid user simulation must respect a strict information-access boundary: a real person only knows their own problem description (the "backstory") and general knowledge—they don't know what terms are inside the search results before they actually search.
LLMs suffer from a "curse of knowledge." Drawing from their massive training data or contextual prompts, they frequently inject highly specific, document-salient terms into initial queries. Traditional evaluation metrics (like lexical overlap or diversity) fail to diagnose this because they cannot distinguish between a rare, creative human query ("human-tail variation") and an LLM query that is effectively telepathic ("answer-side intrusion"). This boundary violation distorts IR system evaluations.
Methods
The authors introduce the Concept Provenance framework to track the origin of semantic units (concepts) within a query and categorize them into priority-ordered zones:
- Z0 (Backstory-supported): Concepts explicitly present in or paraphrased from the user's initial task description.
- Z1 (Human-central): Concepts commonly used by real human searchers ($\ge 10\%$ prevalence).
- Z2 (Human-tail): Rare concepts used by humans ($< 10\%$ prevalence).
- G (Generic): Stopwords or low-importance, non-discriminative terms.
- Z3auto (Candidate answer-side): Concepts completely absent from human queries and the backstory, but highly salient within relevant target documents.
Using this framework, they define the Hidden Concept Intrusion Rate (HCIR) as the proportion of a query's non-generic concepts that fall into the answer-side zone ($Z3_{auto}$).
Datasets
The experiments were conducted using:
- UQV100: An IR test collection containing 100 search topics, complete with backstories and an average of 108 crowdsourced human query variants per topic (10,835 total human queries).
- ClueWeb12-B13: The document collection used for retrieval, containing over 54,000 relevance judgments.
- Synthetic Queries: A newly generated dataset of 77,004 queries produced by 8 state-of-the-art LLMs (from OpenAI, Anthropic, and DeepSeek) across 5 distinct prompting conditions (Generic, Boundary-constrained, High-knowledge, Oracle-text, and Oracle-terms).
Key Insights and Ideas
- The Dual Nature of Intrusion: Human validation revealed that answer-side leakage happens via two distinct mechanisms. First is knowledge intrusion (45.5%), where the LLM uses technical or medical terms a normal user simply wouldn't know. Second is deployment intrusion (45.0%), where the LLM uses common words that a human might know, but wouldn't realistically think to deploy in an initial query before seeing search results.
- LLM Knowledge-Projection Bias: The study proves that LLMs make terrible judges for their own simulation boundaries. When asked to evaluate whether a concept was common knowledge, LLMs consistently exhibited a bias of assuming "if I know this concept, then a typical human pre-search user knows it too," resulting in poor agreement with human annotators.
- Prompts are Leaky Buckets: Even when LLMs are explicitly instructed not to use external or unsupported knowledge (boundary-constrained prompting), they still leak answer-side concepts. One in five queries under the best prompting conditions still contained an intrusive concept.
Key Results
- Prevalence of Cheating: Candidate answer-side concepts ($Z3_{auto}$) make up 7.40% of all non-generic concepts generated by LLMs, appearing in 97 out of 100 search topics.
- Unnatural Retrieval Signals: Through diagnostic deletion probes, the authors showed that removing these $Z3_{auto}$ concepts caused a massive drop in retrieval performance (effect size $d = -0.47$), proving these leaked concepts provide an artificial advantage to the query.
- The Solution (Post-Generation Filtering): While prompt engineering failed to eliminate intrusion, applying the framework as a post-generation filter (Concept-Provenance-Constrained Selection) successfully achieved 99% elimination of answer-side intrusion while maintaining high query utility.
Limitations and Future Work
- Dependency on Dense Human Baselines: Mapping the boundary between a rare human query ($Z2$) and an answer-side leak ($Z3$) requires a high density of human query variants per topic (like UQV100). Applying this framework to datasets without rich human query logs remains a challenge.
- Limited to Initial Queries: The framework currently only evaluates the transition from a backstory to an initial query ($B_t \rightarrow q$). Real-world search involves multi-turn sessions and reformulations where answer-side knowledge is naturally acquired; extending concept provenance to session-level simulation is an open area for future research.
Summary updated