In the burgeoning field of "silicon sampling," researchers use Large Language Models (LLMs) to simulate thousands of survey respondents at a fraction of the cost of human panels. You can prompt a model to act like a "47-year-old forklift operator who knows nothing about dietary supplements" and have it fill out a questionnaire. But there’s a catch: the underlying model does know about supplements from its massive training data.
A recent paper from researchers at Dalpha Inc. asks a critical question: Can an LLM truly "forget" its internal knowledge to play a persona accurately? Their findings suggest the answer is a resounding "no," revealing a phenomenon they call knowledge leakage.
TL;DR
LLMs used for user simulation struggle to suppress their internal knowledge, even when explicitly prompted to play an ignorant persona. This "knowledge leakage" creates a measurable bias where models favor real-world items over fabricated ones, meaning synthetic surveys may reflect the model's training data rather than the intended persona's perspective.
What Problem Does This Paper Address?
The "Persona Premise" in user simulation assumes that a model-driven persona will answer based only on its described characteristics and knowledge. However, LLMs are "played" by a single underlying model that holds documented facts about almost everything.
If a persona is written to be ignorant of a product, but the model knows that product is highly rated, does that knowledge "leak" into the persona's response? This is knowledge leakage: a response bias that arises when information the real-world user could not possibly have enters the simulated response. Measuring this is difficult because, in subjective surveys, there is no "correct" answer to check against.
Methods
To solve the measurement problem, the authors developed the Impostor-Paired Contrast (IPC).
- Fabricated Twins: For every real item (e.g., "Psyllium husk"), they created a "twin" (e.g., "Psyllyer humb")—a nonce term that looks like a real word but doesn't exist.
- Screening: They ensured the models themselves claimed not to recognize these twins in a pre-run screening.
- The Experiment: They prompted LLMs to adopt low-knowledge personas and rate both real items and their twins on a 1–7 scale.
- Knowledge Handling: They tested three ways to enforce ignorance:
- Explicit Ignorance: Telling the model in the prompt, "You have never heard of [Item]."
- Gate-Before: Asking "Have you heard of this?" before the rating.
- Gate-After: Asking the same question after the rating.
If the model were truly ignoring its knowledge, the real item and its twin should receive identical average ratings. Any gap (the IPC) indicates leakage.
Datasets
The researchers introduced a new battery of 404 items across six domains: supplements, skincare, food additives, household chemicals, brands, and civic organizations.
- 178 valenced real items: Items with documented "favorable" or "unfavorable" facts (e.g., a supplement with proven benefits vs. a household chemical with documented hazards).
- 178 screened twins: Matched in surface form and morphology to the real items.
- Reference Sources: Facts were verified via Wikipedia, Wikidata, and regulatory sources.
Key Insights and Ideas
- Self-Reports are Unreliable: When told they haven't heard of an item, models comply perfectly in words—they will say "No, I don't know it." However, their subsequent ratings still show a bias toward the real item. Behavior and self-report disagree.
- Familiarity vs. Content: Leakage isn't just about recognizing a word. It's about the valence of the knowledge. Favorable items get a "familiarity boost," while unfavorable items show more complex behavior.
- The "Hidden" Unfavorable Contrast: Initially, the leakage for unfavorable items looked like zero. However, the authors discovered this was because two opposing biases were canceling each other out. When asked about a "hazard" directly, the model leaked the negative fact (rating the item lower). When asked about "purchase intent" for a brand, the model leaked familiarity (rating the known brand higher despite its flaws).
Key Results
- Persistent Bias: For favorable items, real items were rated +0.324 points higher than their twins under explicit ignorance. This effect held across all five tested API endpoints (including GPT-4 and Gemini families).
- Domain Sensitivity: In domains like "household chemicals," where the scale asked about hazards, the contrast was -0.359 (the model "knew" it was dangerous). In "brands," where the scale was indirect, the contrast was +0.631 (familiarity won out).
- Consistency: The bias wasn't driven by a few "outlier" items; 72 of the 88 favorable items were rated higher than their twins.
Limitations and Future Work
The authors acknowledge several boundaries to their study:
- No Human Baseline: We don't know how a genuinely ignorant human would rate these items, making it hard to say exactly how much the "survey bias" shifts in real-world terms.
- Circular Screening: The twins were screened using the same models being tested.
- Commercial APIs Only: The study focused on closed-weight models (GPT, Gemini); open-weight models might behave differently.
- Single Twin: Using only one twin per item leaves the results susceptible to the specific lexical quirks of that one string.
Future work could involve "low-knowledge human calibration" to provide a true baseline for what an ignorant response looks like. For now, the IPC serves as a vital "model-internal diagnostic" for anyone using LLMs to simulate human populations.
Summary updated