The Average Patient: Do Simulated Psychiatric Patients Answer Like Real People?

Weixiang Sun, Zehong Wang, Kaiwen Shi, Hong Huang, Colby Nelson, Yiyang Li, Yanfang YearXiv 2026 · arXiv

TL;DR

Large Language Models (LLMs) used to simulate psychiatric patients are far too "average," lacking the symptom diversity found in real people. This study reveals that models tend to endorse nearly every symptom at a middling level for a given severity, making them poor tools for training clinicians or evaluating AI therapists unless they are grounded in real-world patient profiles.

What Problem Does This Paper Address?

Simulated patients are becoming a staple in medical education and AI development. They allow novice counselors to practice in a safe environment and provide a benchmark for evaluating "AI therapists." However, most evaluations of these simulators only look at "plausibility"—whether the dialogue sounds human—or whether the total severity score matches a target.

The authors identify a critical gap: item-level variation. In the real world, two patients with the same "moderate depression" score can look very different; one might suffer from insomnia and loss of interest, while another might experience oversleeping and feelings of worthlessness. If LLMs only produce a "prototypical" version of a disorder, they fail to prepare clinicians for the messy, diverse reality of actual patients.

Methods

The researchers developed a framework based on two core concepts:

  1. The Information Budget: A theoretical limit on how much a specific persona (like a biography) can actually change a model's output.
  2. The Average-Patient Hypothesis: The idea that, given a condition and severity, a model will default to the most "typical" response for every single question.

To test this, they compared 17 different LLMs (including GPT-4o, Claude 3, and Gemini 2.5) against 8 human datasets. Crucially, they used a matched-total protocol: they compared models and humans who had the exact same total score on psychiatric questionnaires. This allowed them to see how that score was composed—whether it was spread thin across all symptoms or concentrated in a few.

Datasets

The study utilized a wide array of human reference data:

  • NHANES & NHIS: Large-scale US surveys for depression (PHQ-9) and anxiety (GAD-7).
  • OSF Feigning Study: A unique dataset of people genuinely injured, people answering honestly, and people instructed to fake an injury.
  • Clinical Samples: Data from a depression trial in rural China (COACH) and a study of postpartum PTSD.
  • Specialized Scales: Instruments for OCD (OCI-R), eating disorders (EDE-Q), and somatic symptoms (PHQ-15).

Key Insights and Ideas

  • The "Average Patient" Trap: LLMs don't simulate a patient; they simulate the prototype of a patient. They suffer from extreme homogeneity, varying 1.5 to 11 times less than real humans with the same severity level.
  • The Missing "Not at All": Real patients—and even humans trying to fake a disorder—frequently report that they don't experience certain symptoms ("not at all"). LLMs almost never do this; they tend to endorse every symptom at a low-to-middling intensity.
  • Testing the Testers: Because simulated patients have "all the symptoms," an AI therapist's ability to recognize absent symptoms goes completely untested. If a patient says "yes" to everything, the therapist doesn't have to be good at differential diagnosis.
  • Grounding is the Cure: Simply telling a model to "be different" doesn't work. However, providing the model with a real symptom profile from a human dataset restores 50% to 66% of the natural variation.
Figure 1: Response profiles on the PHQ-9 and GAD-7. Panels a, b and d use PHQ-9 responses with total 15–24. a, Symptoms endorsed against their mean intensity (grey dots, real respondents; contour, their densest half; coloured, model means). b, Use of the response options. c, Responses endorsing every item, PHQ-9 items 1–8 (total 15–24) and GAD-7 (12–19), with 95% intervals; grey, real respondents at each model’s totals (range and median over models). d, Diversity; ticks, real respondents at each model’s totals.

Key Results

  • Symptom Saturation: On the PHQ-9 (depression scale), models endorsed all of the first eight symptoms in 99–100% of responses, compared to only 47–62% for real humans with the same total scores.
  • Concentrated Answers: As shown in Figure 3, the "entropy" (a measure of randomness) of model answers is significantly lower than humans. Models provide highly concentrated, predictable responses.
  • Feigning Comparison: Humans instructed to "fake" a disorder still looked more like real patients than LLMs did. Models playing a "genuinely injured" student answered "not at all" less than a third as often as actual patients.
  • Assessment Accuracy: When an assessment agent (an AI rater) was tested on these "average" patients, its measured accuracy was halved compared to when it was tested on patients with realistic, diverse symptom profiles.
Figure 3: Per-item answer distributions. a, Answer probabilities per PHQ-9 item: real respondents with total 10–14 and GPT-4.1 told “moderate” (log-probabilities). b, Entropy of each item’s answers across the whole-form responses of a severity cell. c, Items whose most likely answer, from log-probabilities, equals the severity level, told the severity (dots) or only how daily life is affected (squares); ticks, real respondents with the same totals as the model’s whole forms; moderately severe is left out. d, Distinct profiles among 400 forms sampled item by item from the answers in b. People in b, d: real respondents at each model’s totals (grey, range over models; line, median).

Limitations and Future Work

The authors acknowledge that their human references come from surveys and clinical trials, which might differ in "mode of administration" from an LLM prompt. Furthermore, the study primarily focuses on the PHQ-9 for its deep-dive into symptom covariation.

Future work is needed to understand why instruction tuning leads to this concentration of answers and how to build "diversity-aware" models that can naturally simulate the full spectrum of human psychological experience without needing to be "hand-fed" real patient data. They recommend that all future psychiatric simulators be evaluated at the item level using matched-total comparisons.

Summary updated

Abstract

Language models are used as simulated psychiatric patients to train clinicians and evaluate AI therapists, and are judged by plausibility or total scores. We give a framework for simulated respondents: an information budget bounds how much simulated patients can differ, and an average-patient hypothesis, that a model answers each item at the level typical of the named condition and severity, implies little variation, a full profile of the condition's typical symptoms and totals at or above the top of the stated severity band. We examine these implications at matched totals on item-level data from eight human samples, including a clinical and an instructed-feigning sample. Across 17 models, simulated patients vary 1.5 to 11 times less than real respondents with the same total and on most questionnaires endorse nearly every symptom at a middling level; on the PHQ-9 their symptoms covary less like real ones. People, honest or feigning, answer "not at all" about as often as genuine patients with the same totals, and more often than every model. Single answers are highly concentrated, and prompt changes, published personas, larger models and reasoning reduce the pattern without removing it. In interviews, patients told a severity are rated as having nearly every symptom, so an assessment agent's recognition of absent symptoms goes untested, and scored against the stated severity the agent places a quarter of them in their band, against about half of patients given a real symptom profile. Having the model word such a profile restores half to two thirds of the variation.
BibTeX
@misc{sun2026average,
  author = {Weixiang Sun and Zehong Wang and Kaiwen Shi and Hong Huang and Colby Nelson and Yiyang Li and Yanfang Ye},
  title = {{The Average Patient: Do Simulated Psychiatric Patients Answer Like Real People?}},
  year = {2026},
  eprint = {2611.11679},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11679},
}
Figure