TL;DR
Large Language Models (LLMs) used to simulate psychiatric patients are far too "average," lacking the symptom diversity found in real people. This study reveals that models tend to endorse nearly every symptom at a middling level for a given severity, making them poor tools for training clinicians or evaluating AI therapists unless they are grounded in real-world patient profiles.
What Problem Does This Paper Address?
Simulated patients are becoming a staple in medical education and AI development. They allow novice counselors to practice in a safe environment and provide a benchmark for evaluating "AI therapists." However, most evaluations of these simulators only look at "plausibility"—whether the dialogue sounds human—or whether the total severity score matches a target.
The authors identify a critical gap: item-level variation. In the real world, two patients with the same "moderate depression" score can look very different; one might suffer from insomnia and loss of interest, while another might experience oversleeping and feelings of worthlessness. If LLMs only produce a "prototypical" version of a disorder, they fail to prepare clinicians for the messy, diverse reality of actual patients.
Methods
The researchers developed a framework based on two core concepts:
- The Information Budget: A theoretical limit on how much a specific persona (like a biography) can actually change a model's output.
- The Average-Patient Hypothesis: The idea that, given a condition and severity, a model will default to the most "typical" response for every single question.
To test this, they compared 17 different LLMs (including GPT-4o, Claude 3, and Gemini 2.5) against 8 human datasets. Crucially, they used a matched-total protocol: they compared models and humans who had the exact same total score on psychiatric questionnaires. This allowed them to see how that score was composed—whether it was spread thin across all symptoms or concentrated in a few.
Datasets
The study utilized a wide array of human reference data:
- NHANES & NHIS: Large-scale US surveys for depression (PHQ-9) and anxiety (GAD-7).
- OSF Feigning Study: A unique dataset of people genuinely injured, people answering honestly, and people instructed to fake an injury.
- Clinical Samples: Data from a depression trial in rural China (COACH) and a study of postpartum PTSD.
- Specialized Scales: Instruments for OCD (OCI-R), eating disorders (EDE-Q), and somatic symptoms (PHQ-15).
Key Insights and Ideas
- The "Average Patient" Trap: LLMs don't simulate a patient; they simulate the prototype of a patient. They suffer from extreme homogeneity, varying 1.5 to 11 times less than real humans with the same severity level.
- The Missing "Not at All": Real patients—and even humans trying to fake a disorder—frequently report that they don't experience certain symptoms ("not at all"). LLMs almost never do this; they tend to endorse every symptom at a low-to-middling intensity.
- Testing the Testers: Because simulated patients have "all the symptoms," an AI therapist's ability to recognize absent symptoms goes completely untested. If a patient says "yes" to everything, the therapist doesn't have to be good at differential diagnosis.
- Grounding is the Cure: Simply telling a model to "be different" doesn't work. However, providing the model with a real symptom profile from a human dataset restores 50% to 66% of the natural variation.
Key Results
- Symptom Saturation: On the PHQ-9 (depression scale), models endorsed all of the first eight symptoms in 99–100% of responses, compared to only 47–62% for real humans with the same total scores.
- Concentrated Answers: As shown in Figure 3, the "entropy" (a measure of randomness) of model answers is significantly lower than humans. Models provide highly concentrated, predictable responses.
- Feigning Comparison: Humans instructed to "fake" a disorder still looked more like real patients than LLMs did. Models playing a "genuinely injured" student answered "not at all" less than a third as often as actual patients.
- Assessment Accuracy: When an assessment agent (an AI rater) was tested on these "average" patients, its measured accuracy was halved compared to when it was tested on patients with realistic, diverse symptom profiles.
Limitations and Future Work
The authors acknowledge that their human references come from surveys and clinical trials, which might differ in "mode of administration" from an LLM prompt. Furthermore, the study primarily focuses on the PHQ-9 for its deep-dive into symptom covariation.
Future work is needed to understand why instruction tuning leads to this concentration of answers and how to build "diversity-aware" models that can naturally simulate the full spectrum of human psychological experience without needing to be "hand-fed" real patient data. They recommend that all future psychiatric simulators be evaluated at the item level using matched-total comparisons.
Summary updated