Abstract: Large language models (LLMs) are trained from vast repositories of text authored by millions of distinct authors, reflecting an enormous diversity of human traits. While these models bear the potential to be used as approximations of human subjects in behavioral studies, prior efforts have been limited in steering model responses to match individual human users. In this work, we introduce "Anthology", a method for conditioning LLMs to particular virtual personas by harnessing open-ended life narratives, which we refer to as "backstories." We show that our methodology enhances the consistency and reliability of experimental outcomes while ensuring better representation of diverse sub-populations. Across three nationally representative human surveys conducted as part of Pew Research Center's American Trends Panel (ATP), we demonstrate that Anthology achieves up to 18% improvement in matching the response distributions of human respondents and 27% improvement in consistency metrics.
Introduction
When interacting with modern Large Language Models (LLMs), users typically encounter a "helpful assistant"—a homogenized voice derived from massive training datasets. While useful for general tasks, this averaged persona is insufficient for social scientists, market researchers, and psychologists who require simulations of diverse, specific, and often contradictory human voices.
Existing methods for "steering" LLMs into specific personas often rely on providing a list of demographic attributes (e.g., "You are a 30-year-old teacher"). However, real human behavior is driven by more than bullet points; it is shaped by life stories, values, and internal narratives. Without this "narrative glue," LLMs often fail to represent underrepresented groups accurately or provide inconsistent answers across a single survey.
The Anthology Methodology
To address these limitations, we propose Anthology, a framework that uses AI-generated, long-form life narratives to create more realistic and consistent virtual personas. The process follows a four-step pipeline designed to bridge the gap between model training data and real-world human distributions.
- Backstory Generation: We prompt a base LLM (e.g.,
davinci-002) with an open-ended question: "Tell me about yourself." This generates a naturalistic first-person narrative covering the persona's upbringing, career, and beliefs. - Demographic Surveying: Since these stories are open-ended, we use a secondary model (GPT-4o) to "interview" these personas, extracting structured demographic data such as age, income, and education.
- Bipartite Matching: To simulate a specific target population, we use matching algorithms (like the Hungarian algorithm) to pair real human respondents from a survey with the virtual personas that most closely match their demographic profiles.
- Survey Administration: The matched backstories serve as a context prefix for the LLM. When asked survey questions, the model responds through the lens of that specific life story.
Our research indicates that base models (such as Llama-3-70B) often outperform "chat-tuned" versions for this task. The Reinforcement Learning from Human Feedback (RLHF) process used in chat models tends to collapse diversity into a single, polite persona, making them harder to steer into diverse viewpoints.
Experimental Setup and Datasets
We evaluated the Anthology framework against three nationally representative surveys from the Pew Research Center’s American Trends Panel (ATP):
- Wave 34: Biomedical and food issues (e.g., views on GMOs).
- Wave 92: Political typology and social trends.
- Wave 99: AI and human enhancement.
We also utilized U.S. Census Data to analyze demographic skews and released an open-source anthology of 10,000 generated backstories to facilitate further research.
Key Results and Insights
- Improved Accuracy: Anthology achieved an 18% improvement in matching the response distributions of real human respondents compared to standard demographic prompting.
- Enhanced Consistency: The method showed a 27% improvement in consistency metrics, ensuring that virtual personas behave like stable individuals rather than providing contradictory answers across a survey.
- Narrative Strength: Life stories capture "latent" traits—personality, cultural values, and personal history—that simple attribute lists miss. This leads to stronger conditioning and more authentic representation of diverse subpopulations.
- Bias Correction: LLMs naturally generate biased backstories (often skewing young, male, and highly educated). The matching step is critical for filtering these outputs to reflect actual real-world diversity.
Discussion and Limitations
While Anthology significantly improves the reliability of "Silicon Sampling," it is important to note that LLMs are emulating human-like distributions, not replacing human subjects. These simulations are best used for pilot studies and hypothesis generation. Furthermore, the diversity of virtual personas is ultimately limited by the diversity of the LLM's training data. As we move forward, ethical considerations regarding the generation of highly realistic personas—which could be misused to manufacture fake public consensus—must remain a priority.
Summary updated