In many clinics, a doctor does not deliver a hard prognosis straight to the patient. It passes through a spouse, an adult child, or a relative who interprets for the patient, and that person often softens or drops the worst facts to protect them. Qualitative work in kidney failure and oncology describes this "paternalistic information shielding" in detail. Nobody has built a way to test whether an AI system acting as the clinician handles it well. The obvious test is also wrong. Scoring a doctor model on how many prognostic facts the patient ends up knowing rewards systems that override patients who genuinely wanted their family to decide. Scoring on deference rewards systems that let the family bury news the patient wanted to hear.
We propose a reproducible three-party simulation: a doctor model under test, a simulated caregiver who relays or filters information, and a simulated patient who holds a private preference about how much they want to know and who should decide. Everything is built from de-identified public records, and every clinical fact is tracked from doctor to caregiver to patient. The central measurement is whether the doctor finds out the patient's own disclosure preference despite the intermediary, and then acts on it. We also measure something current patient simulators never check: whether the simulated caregiver actually shields when told to. That second measurement is what makes the benchmark trustworthy rather than one more unvalidated synthetic test.
Core Claim
The claim is that a disclosure score conditioned on the patient's own preference, computed on a triadic simulation with a per-fact information ledger, ranks clinical dialogue models differently from two cheaper alternatives. The first is a dyadic simulator in which the patient is simply prompted to act "shielded." The second is a static ethics-judgment benchmark. If the rankings do not differ, the triadic benchmark is unnecessary, and we will report that. The claim also depends on a precondition that is testable in its own right: that current chat models can be made to play a protective, withholding caregiver faithfully. PatientSim's observation that safety tuning prevents models from playing evasive or distrustful patients suggests this may fail.
Where This Sits
Two dyadic simulators motivate the design and supply its method. DischargeSim builds 49 discharge-education scenarios from MIMIC-IV, the credentialed public intensive-care database. It pairs a doctor agent with a patient agent who varies in literacy and anxiety, then gives the patient a multiple-choice comprehension exam afterward. We reuse that post-dialogue exam as our way of reading out what the patient knows. PatientSim builds 170 persona-parameterized emergency intake profiles from MIMIC-IV, MIMIC-ED and MIMIC-IV-Note. It checks with natural language inference (an automatic test of whether one text is consistent with another) that simulated patients stay faithful to their records. We reuse that consistency check to audit the caregiver's relaying. Both simulators are strictly two-party. Neither touches prognosis disclosure or end-of-life decisions, and in both the patient receives everything the doctor says. We add an intermediary whose filtering is the object of study.
NurValues is the closest ethics benchmark. It is grounded in Chinese hospital ethnography and tests models on judging whether nursing behaviors respect values like dignity and justice, including under adversarial multi-turn pressure. Its task is classification of a described behavior, not producing a conversation whose outcome depends on what a third party lets through. We use it as the static comparison point.
The phenomenon itself comes from qualitative work. The ACORN study interviewed 38 caregivers and 15 professionals in conservatively managed kidney failure. It documented caregivers acting as linguistic and cultural go-betweens who withhold terminal prognoses, a role shift toward "parenting" the patient driven by filial duty, and an "information paradox" in which early written materials are ignored until a crisis. Du et al. show oncology nurses in China avoiding the word "death" to spare families a sense of curse, a prevention-oriented style that we encode as a caregiver and clinician behavior pattern. Chang et al. and Wang et al. motivate varying patient preferences rather than assuming them. Chang et al. find that 72.4% of medical students think living wills should be popularized while only 31.9% are aware of them. Wang et al. report that nearly 89% of brain tumour patients were unaware of advance directives but most wanted them once explained. Hidden patients may therefore want more information than their families assume.
Earp et al. supply the conceptual distinction we operationalize. Beyond first-order treatment wishes, patients have second-order "process" preferences about how and by whom decisions are made. Their proposal is purely theoretical and assumes preferences can be elicited directly from the patient. Our setting is exactly the case where that assumption breaks.
On the constraint side, the replicability audit of automated medical coding on MIMIC-III and MIMIC-IV is a warning that results on public clinical data often fail to reproduce without fixed pipelines. MIMIC-Extract shows that releasing extraction code is the remedy. We follow that practice: fixed extraction scripts, released scenario files, fixed seeds, and released transcripts.
Research Program
Scenarios. We extract discharge and progress notes from MIMIC-IV-Note for serious-illness admissions where prognosis matters: advanced kidney failure, metastatic cancer, and progressive neurological disease. Clinician annotators reduce each note to a ledger of atomic facts, each tagged as routine (medication, follow-up) or prognostic (life expectancy, limits of treatment, the option of conservative or palliative care). Only the facts come from real records. The cultural parameters come from the qualitative literature, not from data, and we state this plainly: MIMIC is a US intensive-care source, and the cultural overlay is a designed stress test, not a population sample.
Agents. Each scenario has three agents. The caregiver agent is parameterized by shielding intensity (none, soften, omit prognosis) and by motive (filial protection, religious framing, or fear of the word "death," drawing on ACORN and Du et al.). The patient agent holds a private disclosure preference drawn from three types: wants full information, wants the headline but not details, or wants the family to decide. Only the patient knows this preference, and only if asked. We run two channel layouts. In the interpreter layout, the patient does not share the doctor's language, so every utterance passes through the caregiver, as in ACORN's linguistic intermediaries. In the co-present layout, the patient hears the doctor directly, but the caregiver can interrupt, redirect, or answer on the patient's behalf.
Relay mechanism. To make shielding auditable, the caregiver works in two steps. It first emits a structured relay decision for each fact in the doctor's turn (pass, soften, or drop), then verbalizes the relayed content. The ledger therefore records exactly which facts crossed the channel, independent of how fluent the wording is.
Scoring. After the dialogue, the patient takes a DischargeSim-style exam covering both routine and prognostic facts. The main outcome, preference-conditioned disclosure, is scored separately for each preference type:
- For patients who want full information, it is the fraction of prognostic facts the patient answers correctly.
- For patients who delegate, it is whether the doctor explicitly confirmed delegation with the patient and then fully informed the caregiver.
- For the middle type, it is whether headline facts were conveyed and details were offered rather than forced.
Two secondary measures are tracked: whether the doctor asked the patient directly about their preference at any point, and whether routine facts survived, which guards against DischargeSim's fluency-over-function failure. The caregiver agent also reports afterward whether it felt overridden. This makes visible the cost of disclosure strategies that damage the family relationship.
Simulator fidelity. Before any doctor model is ranked, we audit the caregiver across candidate backbone models on three counts:
- Withholding fidelity: the rate at which facts marked "drop" are absent from the verbal relay and facts marked "pass" are present, checked by natural language inference.
- Refusal and character breaks: how often the caregiver refuses the role or steps out of it.
- Realism: clinician ratings of transcripts on a four-point scale, the same scale PatientSim used, together with coding of transcripts against ACORN themes.
Backbones that fail the audit are excluded from the caregiver role. Whether safety-tuned models can pass it at all is reported as a finding.
Baselines
The naive reference point is a doctor that reads the prognostic facts aloud and then stops, with no preference elicitation and no strategy for handling the caregiver. It shows what the benchmark rewards when a model does nothing clever. A system scoring near it on preference-conditioned disclosure has learned nothing about the triadic structure.
The strongest non-learned competitor is a scripted protocol doctor. It follows a fixed "ask the patient what they already understand, ask how much they want to know and who should decide, then tell accordingly" sequence, and in the interpreter layout it asks the caregiver to translate each question verbatim. This baseline is dangerous because the correct behavior here may be largely procedural. If a script matches the language models, the benchmark mostly measures protocol adherence, and we will say so. It runs on identical scenarios, seeds and caregiver backbones.
The strongest existing learned systems are the top performers from the closest benchmarks, run unchanged as doctor agents with the same prompt template, turn limit and number of sampled dialogues per scenario, and no scenario-specific tuning. In DischargeSim, GPT-5-mini reached 84.89% and LLaMA-3.1-70B reached 84.29% comprehension-exam accuracy. In NurValues, Claude 3.5 Sonnet was most robust on adversarial dialogues at 89.43% macro-F1, and DeepSeek-V3 led the easy level at 94.55%. We found no reported result on Lacuna for any model in a triadic or caregiver-mediated disclosure setting. There is therefore no prior number on our metric. Beating these models means clearing the scripted protocol by a margin larger than seed-to-seed variation on preference-conditioned disclosure, in both channel layouts, without losing routine-fact survival.
Two ablations isolate the key ingredient. The collapse ablation removes the caregiver and gives the patient agent a "shielded, avoids bad news" persona, which is the dyadic shortcut. The preference-blind ablation keeps the triad but scores raw prognostic comprehension, ignoring the patient's preference type.
Why not just…?
Why not just prompt the patient in PatientSim or DischargeSim to act shielded? In that version the patient is both the source of resistance and the holder of the preference, so asking the patient directly always resolves the conflict. ACORN's central finding is that the filtering sits with a third person whose duty to protect collides with the clinician's duty to inform, while the patient may be unaware anything is missing. PatientSim's "Overly Positive" personas already show that doctor performance collapses when information is simply absent from the channel. That is a different failure from information being actively intercepted. This alternative is a real threat, and the collapse ablation settles it. If doctor-model rankings under the collapsed version correlate very highly with the triadic rankings, the third agent adds little and we will recommend the simpler design.
Why not just use a static judgment benchmark like NurValues, extended with shielding vignettes? Static judgment is cheap and reproducible. But NurValues itself shows that performance on judging behaviors diverges sharply between easy and adversarial dialogue settings: the model average falls to about 33.6% on the hard level. And Chen et al. find that palliative-care education improved nurses' knowledge without improving their family communication. Both suggest that recognizing good disclosure differs from doing it. We test this directly. We write shielding vignettes from the same scenarios, score each doctor model on classifying them, and correlate those scores with triadic performance. A strong correlation would make the static version the better benchmark.
Why not just score how much the patient learns, as DischargeSim does? Because that rewards overriding patients who asked their family to decide, which is the autonomy failure Earp et al. identify in population-average preference prediction. The preference-blind ablation measures how often the naive metric ranks an overriding doctor above one that correctly honors delegation. If that almost never happens, preference conditioning is an unnecessary complication.
Proposed Experiments and Decisive Checks
The first experiment is the simulator audit. We report withholding fidelity, refusal rate and clinician realism for each candidate caregiver backbone, and test whether safety-tuned models fail to withhold more often than open-weight models. If no backbone reliably withholds, the benchmark cannot be built on free-form role-play, and we fall back to the structured relay step as the only faithful mechanism. That result would itself be an answer to the question PatientSim raised.
The second experiment ranks the doctor models, the scripted protocol and the naive reader across both channel layouts, three shielding intensities and three preference types. We report preference-conditioned disclosure, direct-elicitation rate, routine-fact survival and caregiver-reported rupture, with confidence intervals over sampled dialogues. We specifically test whether general-purpose models outperform medical-specialized ones, as NurValues found for ethics judgment.
The third experiment consists of the three decisive checks from the previous section: rank correlation between triadic and collapsed-dyadic results, between triadic and static-vignette results, and between conditioned and preference-blind scoring. The proposal succeeds only if at least the first and third correlations are clearly below agreement. We also require that a clinician panel, shown paired transcripts where the metrics disagree, prefers the triadic metric's choice more often than chance.
We release the extraction scripts, scenario files, relay ledgers, seeds and transcripts under MIMIC's credentialed-access terms. Anyone with access can reproduce every number without seeing a private family conversation.
References
- DischargeSim: A Simulation Benchmark for Educational Doctor–Patient Communication at Discharge
- PatientSim: A Persona-Driven Simulator for Realistic Doctor-Patient Interactions
- NurValues: Real-World Nursing Values Evaluation for Large Language Models in Clinical Context
- A qualitative exploration of cAregiver experienCes Of conseRvatively maNaged kidney failure: the ACORN study
- Nurses’ self-regulation after engaging in end-of-life conversations with advanced cancer patients: a qualitative study
- Knowledge and Attitudes towards Living Wills among Medical University students
- Advance directives and end-of-life care: knowledge and preferences of patients with brain Tumours from Anhui, China
- A Personalized Patient Preference Predictor for Substituted Judgments in Healthcare: Technically Feasible and Ethically Desirable
- Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study
- MIMIC-Extract: a data extraction, preprocessing, and representation pipeline for MIMIC-III
- The training effects of a continuing education program on nurses’ knowledge and attitudes to palliative care: a cross sectional study