Assessing the Human-Likeness of LLM-Driven Digital Twins in Simulating Health Care System Trust

Yuzhou Wu, Mingyang Wu, Di Liu, Rong Yin, Kang LiarXiv 2025 · DOI · arXiv

Abstract

This study evaluates the capacity of Large Language Models (LLMs) to simulate complex human psychological traits, specifically healthcare system trust. By generating "digital twins" from detailed personas and prompting ChatGPT-4 to complete the Health Care System Distrust Scale (HCSDS), researchers compared AI-generated responses against a real-world human reference group. The findings reveal a significant "variance collapse" where LLMs avoid extreme opinions, clustering instead around moderate, "safe" responses. This tendency results in an underestimation of human skepticism and distrust, suggesting that current LLMs require extensive calibration before they can reliably model polarized human attitudes in healthcare policy simulations.

Introduction

As Large Language Models (LLMs) advance, they are increasingly utilized to create "Human Digital Twins"—virtual personas designed to simulate the behaviors and attitudes of specific demographic groups. In healthcare systems engineering, these twins offer a potential sandbox for testing new policies or communication strategies. However, the utility of these simulations depends on their ability to capture the nuances of human psychology, particularly trust. Healthcare system trust is a multi-faceted trait influenced by historical context, race, and socioeconomic status. If AI simulations fail to capture the depth of community distrust, the resulting policies could be ineffective or harmful. This research investigates the gap between LLM-simulated attitudes and actual human-reported data.

Methods

The study employed a multi-step simulation pipeline:

  1. Persona Construction: Detailed profiles of 2,058 U.S. adults were drawn from the Twin-2K-500 dataset, encompassing over 500 features including demographics and behavioral history.
  2. LLM Prompting: These personas were used as prompts for ChatGPT-4, which was tasked with completing the 10-item Health Care System Distrust Scale (HCSDS) as if it were the described individual.
  3. Comparative Analysis: A 500-case subsample was created using stratified random sampling to match the demographics of a reference group of 400 Philadelphia jurors from a seminal 2004 study.
  4. Statistical Validation: Distributions were compared using Chi-square tests for alignment and Cohen’s $d$ for effect sizes.

Results

The Variance Collapse Phenomenon

The most prominent finding was the "variance collapse" in LLM responses. While human respondents frequently expressed strong or extreme views (selecting 1 or 5 on the Likert scale), the LLM-driven twins consistently clustered around the center. As shown in Figure 1, the AI-generated data exhibits "taller peaks" at moderate values, failing to replicate the "long tails" of human opinion.

Distribution of responses for HCSDS items Q1-Q10, comparing LLM-driven digital twins to human reference data.

Statistical Divergence and Effect Sizes

Chi-square tests revealed highly significant differences ($p < 0.001$) between LLM and human responses for all 10 scale items. The LLM consistently simulated lower levels of distrust than the human sample. As detailed in Figure 2, for items probing whether the healthcare system prioritizes medical needs above all else, the effect sizes were massive (Cohen’s $d$ of -1.44 and -1.38), indicating a substantial underestimation of human skepticism.

Mean scores, standard deviations, and Cohen’s d effect sizes for the 10 HCSDS items.

Demographic and Bias Observations

While the LLM successfully reproduced broad demographic patterns related to age and gender, it struggled with more subtle correlations, such as the relationship between education levels and distrust. Furthermore, the researchers hypothesize that the LLM's "safety filters"—designed to ensure helpfulness and harmlessness—may inadvertently suppress the simulation of controversial or extreme attitudes, leading to a "mainstream narrative" bias.

Discussion and Limitations

The study highlights that LLMs cannot currently be used "out of the box" for high-stakes social simulations. Several limitations were noted, including a temporal mismatch between the 2004 reference data and the modern persona dataset. Additionally, the subsample had a higher proportion of college-educated individuals (83%) compared to the reference group (70%), which may have influenced the results.

Future research should explore Item Response Theory (IRT) and multi-model benchmarking to better understand the "emotional reasoning" of LLMs. Effective deployment in healthcare policy will require rigorous calibration, importance reweighting, and manual oversight to bridge the gap between AI-simulated personas and real-world human complexity.

Summary updated

BibTeX
@misc{wu2025assessing,
  author = {Yuzhou Wu and Mingyang Wu and Di Liu and Rong Yin and Kang Li},
  title = {{Assessing the Human-Likeness of LLM-Driven Digital Twins in Simulating Health Care System Trust}},
  year = {2025},
  eprint = {2512.08939},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.08939},
}
Figure