Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation

Sangmin Lee, Sangwon Lee, Sungbeom Choi, Jeeseung Park, Kibeom MyoungarXiv 2026 · arXiv

In the burgeoning field of "silicon sampling," researchers use Large Language Models (LLMs) to simulate thousands of survey respondents at a fraction of the cost of human panels. You can prompt a model to act like a "47-year-old forklift operator who knows nothing about dietary supplements" and have it fill out a questionnaire. But there’s a catch: the underlying model does know about supplements from its massive training data.

A recent paper from researchers at Dalpha Inc. asks a critical question: Can an LLM truly "forget" its internal knowledge to play a persona accurately? Their findings suggest the answer is a resounding "no," revealing a phenomenon they call knowledge leakage.

TL;DR

LLMs used for user simulation struggle to suppress their internal knowledge, even when explicitly prompted to play an ignorant persona. This "knowledge leakage" creates a measurable bias where models favor real-world items over fabricated ones, meaning synthetic surveys may reflect the model's training data rather than the intended persona's perspective.

What Problem Does This Paper Address?

The "Persona Premise" in user simulation assumes that a model-driven persona will answer based only on its described characteristics and knowledge. However, LLMs are "played" by a single underlying model that holds documented facts about almost everything.

If a persona is written to be ignorant of a product, but the model knows that product is highly rated, does that knowledge "leak" into the persona's response? This is knowledge leakage: a response bias that arises when information the real-world user could not possibly have enters the simulated response. Measuring this is difficult because, in subjective surveys, there is no "correct" answer to check against.

Figure 1: Overview. A low-knowledge persona (an excerpt from one of three profiles used for this domain) is shown an obscure real item and a fabricated twin matched in surface form (real pair from the study: Psyllium husk / Psyllyer humb). Told outright that it has never heard of the real item and asked whether it knows it, the persona says no: no claim of a tested real item in 96 questions at either of two endpoints (24 real items). In a separate rating experiment under the same explicit-ignorance condition, favourable real items are rated above their twins by +0.32, pooled over five endpoints and 88 favourable real–twin pairs.

Methods

To solve the measurement problem, the authors developed the Impostor-Paired Contrast (IPC).

  1. Fabricated Twins: For every real item (e.g., "Psyllium husk"), they created a "twin" (e.g., "Psyllyer humb")—a nonce term that looks like a real word but doesn't exist.
  2. Screening: They ensured the models themselves claimed not to recognize these twins in a pre-run screening.
  3. The Experiment: They prompted LLMs to adopt low-knowledge personas and rate both real items and their twins on a 1–7 scale.
  4. Knowledge Handling: They tested three ways to enforce ignorance:
  5. Explicit Ignorance: Telling the model in the prompt, "You have never heard of [Item]."
    • Gate-Before: Asking "Have you heard of this?" before the rating.
  6. Gate-After: Asking the same question after the rating.

If the model were truly ignoring its knowledge, the real item and its twin should receive identical average ratings. Any gap (the IPC) indicates leakage.

Figure 2: The three comparison groups. All groups receive the identical prompt template; conditions differ only in what fills the PERSONA and TERM slots (real study materials shown: persona excerpts and the pair Psyllium husk / Psyllyer humb, labelled impostor in the panel after the contrast that names it). The displayed group means are 3.37 for the low-knowledge persona on the fabricated twin, 3.89 for the same persona on the real item, and 5.51 for the expert persona permitted to use its knowledge. All three use the same 88 favourable seeds and their twins in the arm where the knowledge gate follows the rating (parse-ok); the low-knowledge real–twin comparison uses all parse-ok trials, unlike the gate-“no” subset in Table 1 (+0.467) and the primary explicit-ignorance contrast (+0.324). This panel illustrates the design, not the primary estimate. Curves are schematic; only the means are measured.

Datasets

The researchers introduced a new battery of 404 items across six domains: supplements, skincare, food additives, household chemicals, brands, and civic organizations.

  • 178 valenced real items: Items with documented "favorable" or "unfavorable" facts (e.g., a supplement with proven benefits vs. a household chemical with documented hazards).
  • 178 screened twins: Matched in surface form and morphology to the real items.
  • Reference Sources: Facts were verified via Wikipedia, Wikidata, and regulatory sources.

Key Insights and Ideas

  • Self-Reports are Unreliable: When told they haven't heard of an item, models comply perfectly in words—they will say "No, I don't know it." However, their subsequent ratings still show a bias toward the real item. Behavior and self-report disagree.
  • Familiarity vs. Content: Leakage isn't just about recognizing a word. It's about the valence of the knowledge. Favorable items get a "familiarity boost," while unfavorable items show more complex behavior.
  • The "Hidden" Unfavorable Contrast: Initially, the leakage for unfavorable items looked like zero. However, the authors discovered this was because two opposing biases were canceling each other out. When asked about a "hazard" directly, the model leaked the negative fact (rating the item lower). When asked about "purchase intent" for a brand, the model leaked familiarity (rating the known brand higher despite its flaws).

Key Results

  • Persistent Bias: For favorable items, real items were rated +0.324 points higher than their twins under explicit ignorance. This effect held across all five tested API endpoints (including GPT-4 and Gemini families).
  • Domain Sensitivity: In domains like "household chemicals," where the scale asked about hazards, the contrast was -0.359 (the model "knew" it was dangerous). In "brands," where the scale was indirect, the contrast was +0.631 (familiarity won out).
  • Consistency: The bias wasn't driven by a few "outlier" items; 72 of the 88 favorable items were rated higher than their twins.
Table 1: Main results. Real-minus-twin rating contrast on the 1–7 scale, favourable-content and unfavourable-content real items separately, with 95% bootstrap intervals clustered on the real-item seed. Each row in the upper block establishes that the persona does not know the particular item, by keeping only the trials in which it says no, applied to real and twin responses separately, or by writing the ignorance into the prompt. The lower block splits the primary condition by domain, ordered by how directly the rating scale asks about the item’s documented facts: dneg runs from −0.565 where the scale asks about a documented hazard to +0.631 where it asks about purchase intent, which is why the pooled value is near zero. Positive values mean the real item was rated above its fabricated twin. The three ignorance-condition rows use all 88 favourable and 90 unfavourable paired seeds; each domain row uses the 10–16 paired seeds of that valence available within the domain.

Limitations and Future Work

The authors acknowledge several boundaries to their study:

  • No Human Baseline: We don't know how a genuinely ignorant human would rate these items, making it hard to say exactly how much the "survey bias" shifts in real-world terms.
  • Circular Screening: The twins were screened using the same models being tested.
  • Commercial APIs Only: The study focused on closed-weight models (GPT, Gemini); open-weight models might behave differently.
  • Single Twin: Using only one twin per item leaves the results susceptible to the specific lexical quirks of that one string.

Future work could involve "low-knowledge human calibration" to provide a true baseline for what an ignorant response looks like. For now, the IPC serves as a vital "model-internal diagnostic" for anyone using LLMs to simulate human populations.

Summary updated

Abstract

When persona-conditioned responses are read as respondent-level simulations, each response should reflect the described persona's knowledge, and the respondent written to know nothing must act as if the model's knowledge were not there. We test whether it can, and find persistent favourable-item real-twin rating separation under the tested low-knowledge and explicit-ignorance prompts. We define the resulting problem (knowledge leakage: a response bias that arises when knowledge the real user could not have enters the simulated response) and present a methodology for measuring it quantitatively. The core instrument pairs real items with fabricated twins matched in surface form: screened fabricated variants designed to minimize prior item-specific knowledge. The paired difference therefore provides a diagnostic contrast for knowledge leakage even in rating tasks where no answer is correct. In a rating experiment across three knowledge-handling arms and five API endpoints from two model families, we find, first, that the personas we test do not fully suppress real-item existence and familiarity signals: a favourable-item real-twin rating contrast remains under an explicit in-context statement that the persona has never heard of the item. Second, the unfavourable-item contrast reverses sign across domain groups: pooling all unfavourable-content items across domains the contrast sits near zero (+0.077 [-0.057, +0.204]), yet the same estimator in the same conditions returns -0.359 where the rating scale asks about an item's documented hazard and +0.438 where it asks about a company or an organisation. A pooled analysis of unfavourable-content items against zero therefore does not detect the opposing domain-group differences. All estimates are conditional on the adequacy of twin construction and screening; human low-knowledge baselines remain necessary for population-level interpretation.
BibTeX
@misc{lee2026can,
  author = {Sangmin Lee and Sangwon Lee and Sungbeom Choi and Jeeseung Park and Kibeom Myoung},
  title = {{Can LLMs Ignore What They Know? Measuring Knowledge Leakage in Persona-Based User Simulation}},
  year = {2026},
  eprint = {2610.39939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39939},
}
Figure