People who fear judgment about depression or suicidal thoughts often deny symptoms when asked directly. They also often refuse to use any tool that sends their words to a company's servers. Recent work shows that a chatbot asking indirect, normalizing questions recovers more diagnoses from such people. Those results, however, come from a cloud model (GPT-4) tested against a few dozen simulated patients played by another large language model. This proposal argues that the main obstacle to a private, on-device version is not the model's size. It is the sameness of the simulated patients used to train and grade it. If every simulated patient deflects in the same polite way, a small model learns a single script, and the evaluation rewards that script.
We propose building a deliberately varied population of simulated stigmatized patients. Each patient keeps a fixed, checkable set of symptoms, but patients differ widely in how they hide, minimize, deflect or partially disclose. We use this population for two jobs. The first is to train a compact interviewer that runs locally. On each turn it chooses both what to ask and how the user may answer: free text, tappable options, a slider, or "skip for now". The second is to test whether simulator diversity changes which interviewing strategies look best. The new claim is specific and falsifiable. Simulator homogeneity inflates and reorders the measured performance of stigma-aware interviewers. Diversifying the defensive behavior of simulated patients, rather than their surface persona, is what lets a small local model match a prompted frontier model on held-out patients and on human role-players.
Where the gap is
The motivating evidence comes from four lines of work that have not been joined.
Breaking the Stigma! Unobtrusively Probe Symptoms in Depression Disorder Diagnosis Dialogue builds UPSD⁴. It pairs a module that picks indirect probing tactics, such as normalizing a symptom before asking about it, with a module that fills diagnostic slots against ICD-11 criteria. On stigmatized personas derived from the D4 Chinese depression-diagnosis dialogues, it raises GPT-4's diagnostic accuracy from 25.00% to 34.09% and its diagnosis completion rate from 38.64% to 59.85%. That work supplies our task and strongest baseline. It also exposes the weakness we target: the patient side is ten stigma profiles played by an LLM, and everything runs on a cloud API.
Two findings say why such simulators are suspect. The Colorful Future of LLMs finds that LLM responses to queer youth cluster narrowly in embedding space around one polite persona, while human responses spread widely. Towards Algorithmic Fidelity finds that LLM-generated depression posts stereotype by demographic and have suicide and self-harm content removed entirely by safety filtering. Neither paper studies patient simulators for interviewing agents, but both predict that LLM-played patients will be both too uniform and too sanitized.
Three papers supply methods.
- Diverse generation. CorrSynth generates several samples in parallel while penalizing them for picking the same tokens. It improves student classifiers, for example TOI Headlines accuracy from 69.7% to 71.1%. It has only been applied to single-label text classification. We apply it to the patient's turn inside a multi-turn dialogue, contrasting replies that share the same hidden symptoms.
- Measuring diversity. Measuring Diversity in Synthetic Datasets gives DCScore, a cheap embedding-based diversity measure, which we use to quantify simulator coverage.
- Specialization. Specializing Smaller Language Models towards Multi-Step Reasoning and Lightweight Transformers for Conversational AI show that small models trained narrowly can rival much larger ones, at the cost of general ability. We reuse that recipe.
One finding constrains the design. An Interpretability-Guided Framework for Responsible Synthetic Data Generation in Emotional Text shows that steering generation toward discriminative keywords raises classifier scores but strips personal framing such as pronoun-verb patterns. Our diversity therefore must not come from making symptoms more legible.
The interface component borrows from Decision-Theoretic User Interface Generation. Its SUPPLE system picks widgets to minimize a per-user cost. That approach yielded 26% faster task completion and 73% fewer errors for motor-impaired users in Wobbrock et al.. We replace motor cost with the cost of disclosing: some users will tap "sometimes" on a slider who would never type "I have thought about dying". This is an extension of the agent's action space, not the core claim.
Core Claim
Holding the interviewer fixed, scores measured against homogeneous LLM-played patients overestimate performance on human role-players, and they rank strategies differently. Training a compact interviewer on dialogues with patients diversified in defensive behavior closes that gap. The comparison is against equal-sized training sets from persona-prompted or high-temperature simulators.
Research Program
Patient profiles. Each simulated patient is a profile with two parts. The hidden clinical part is a symptom set and a severity drawn from D4's clinical portraits; it serves as ground truth. The defensive part describes behavior, including:
- denial, minimization, humor, irritation;
- deflecting onto physical complaints;
- disclosing only after a normalizing statement;
- disclosing only through a structured option;
- cultural and family context that shapes what is unsafe to say.
Generating diverse patient replies. The patient is played by an open-weight model run locally. Running locally gives the logit access CorrSynth needs and avoids commercial filtering of self-harm content. At each patient turn we sample K replies in parallel with CorrSynth's anti-correlation penalty, applied among replies that share the same hidden symptoms and the same stigma level. This pushes diversity into how patients defend themselves rather than what is wrong with them.
Fidelity check. A separate symptom-consistency judge rejects any reply that implies a symptom outside the profile, or denies one the profile marks as clearly disclosable. A small clinician-audited sample calibrates this judge. Without this filter, diversity would corrupt the labels.
The interviewer. A teacher is GPT-4 running UPSD⁴-style prompting plus a widget choice on each turn. It interviews the patient population, and successful dialogues, meaning the correct diagnosis is reached without a harmful utterance, are distilled into a compact open-weight model of roughly 1–8B parameters. That model is then quantized for on-device use, following the local-deployment route in Natural Language Explanations for Suicide Risk Classification.
Optional reinforcement-learning step. An optional second stage fine-tunes the interviewer with reinforcement learning against the diverse simulators. The reward is diagnostic slot accuracy minus penalties for:
- turns used;
- the simulated patient withdrawing;
- any utterance flagged as unsafe.
Ethics. Generated high-risk text stays inside the research environment, and any human study uses trained role-players rather than at-risk participants.
Baselines
Do-nothing reference. The do-nothing reference is vanilla prompted GPT-4 interviewing the stigmatized personas. Cao et al. report 25.00% diagnostic accuracy and a 38.64% completion rate for it. We rerun it on our patient pools with the same turn limit, so the comparison does not rest on their simulator alone.
Strongest existing method. The strongest existing method is UPSD⁴-GPT-4, with 34.09% accuracy and 59.85% completion rate reported by Cao et al.. It is the strongest competitor because it is the only system in the file built for defensive, stigmatized patients. We reimplement it from the paper's description and evaluate it on every patient pool and on human role-players. Because our simulators differ from theirs, the reported numbers serve as an anchor, not a target. Beating it means the compact local model matches or exceeds reimplemented UPSD⁴-GPT-4 on human role-players: completion rate within the confidence interval or higher, and accuracy no lower. It must not raise the harmful-utterance rate.
Local prompting without training. A local quantized model prompted with UPSD⁴'s tactics tests whether training is needed at all. This follows the Stern et al. approach, which reached F1 0.41 on the UMD Reddit Suicidality Dataset in a classification setting. No reported result was found on Lacuna for local prompted models on D4-based stigma interviewing.
Key ablation. The ablation that isolates the key ingredient keeps the compact model, the teacher, the training-set size and the tuning budget identical. It varies only the patient pool used to produce training dialogues, comparing three pools:
- UPSD⁴-style ten profiles;
- many persona prompts with high sampling temperature;
- our CorrSynth-diversified, fidelity-filtered pool.
A second ablation removes widget choice from the action space. For reference on question efficiency, we also report an ALBA-style adaptive item-selection screener. That screener reached r > 0.9 with latent depression in three questions on non-stigmatized online participants, but it does not handle defensive denial.
Why not just…?
Why not just diversify simulators with more persona prompts and higher temperature? This is the real threat. The persona-driven augmentation in PersonaMath is the cheap version of our idea. The existing evidence points both ways:
- For our design: Lissak et al. find that prompted LLMs stay clustered around one persona.
- For our design: CorrSynth beats few-shot generation on MAUVE, 94.4 vs 87.1 on AG News.
- Against it: neither result concerns defensive dialogue.
The pool ablation settles it directly. If the persona-prompted pool matches ours in DCScore over defensive replies and in downstream human role-play performance, we will report that and drop the anti-correlated sampling from the recommended recipe.
Why not just distill UPSD⁴-GPT-4 transcripts into a small model? That is exactly our homogeneous-pool condition. Our hypothesis is that it reproduces the teacher on familiar simulators and falls off on held-out patient families and human role-players. Fu et al. show that specialized students lose ability outside their training distribution. Cross-pool transfer, training on one pool and testing on the others and on humans, measures this directly.
Why not just prompt a quantized local model with UPSD⁴'s tactics? Stern et al. find that such models match MentalBERT on suicide-risk classification but invent self-harm details absent from the patient's text. Multi-turn probing compounds that risk. We include this condition as a baseline and track hallucinated-symptom rate alongside accuracy.
Why not just offer an anonymous PHQ-9 form with a skip button? It is the cheapest private option and needs no model. Cao et al.'s gap between direct and indirect questioning, a 38.64% vs 59.85% completion rate, suggests direct items lose defensive users. However, that comparison was between dialogue agents, not forms. We therefore include an anonymous form condition in the human role-play study.
Proposed Experiments
Experiment 1: does the patient pool change the verdict? We test whether simulator homogeneity distorts evaluation. We fix five interviewers:
- vanilla GPT-4;
- UPSD⁴-GPT-4;
- prompted local model;
- homogeneous-distilled compact model;
- diverse-distilled compact model.
We score each on the three patient pools and on a human role-play set. In that set, trained actors play held-out D4 portraits from written stigma scripts they did not see the simulators use. The decisive check compares rank correlation (Kendall's tau) between each pool's interviewer ranking and the human ranking, plus the absolute gap in completion rate. Our claim fails if the homogeneous pool predicts human rankings as well as the diverse one.
Experiment 2: does diverse training data make a small model good? We train the compact interviewer on each pool at matched size and compute, then evaluate on held-out patient families and on the human role-play set. The measures are:
- diagnostic accuracy and completion rate, defined as in Cao et al.;
- turns to diagnosis;
- hallucinated-symptom rate;
- harmful-utterance rate under a fixed safety rubric;
- blinded human ratings of discreetness and empathy.
We also report on-device latency of the quantized model and its loss on a general benchmark, to quantify the specialization cost.
Experiment 3: does choosing the answer format help disclosure? We compare the full interviewer with and without widget choice, measuring symptom slots filled per turn and withdrawal rate. Simulated patients answering structured options is the least validated part of the pipeline. We therefore treat the simulator result as provisional and count only the human role-play comparison as evidence.
References
- Breaking the Stigma! Unobtrusively Probe Symptoms in Depression Disorder Diagnosis Dialogue
- D4: a Chinese Dialogue Dataset for Depression-Diagnosis-Oriented Chat
- The Colorful Future of LLMs: Evaluating and Improving LLMs as Emotional Supporters for Queer Youth
- Towards Algorithmic Fidelity: Mental Health Representation across Demographics in Synthetic vs. Human-generated Data
- CorrSynth - A Correlated Sampling Method for Diverse Dataset Generation from LLMs
- Measuring Diversity in Synthetic Datasets
- Specializing Smaller Language Models towards Multi-Step Reasoning
- Lightweight Transformers for Conversational AI
- An Interpretability-Guided Framework for Responsible Synthetic Data Generation in Emotional Text
- Decision-Theoretic User Interface Generation
- Improving the performance of motor-impaired users with automatically-generated, ability-based interfaces
- Natural Language Explanations for Suicide Risk Classification Using Large Language Models
- ALBA: Adaptive Language-Based Assessments for Mental Health
- PersonaMath: Enhancing Math Reasoning through Persona-Driven Data Augmentation