Clinical records are fragmented. Labs done at an outside hospital, notes stored separately from billing codes, and patients who move between health systems all leave gaps. AI systems that read these records tend to guess when they should say the evidence is missing. On CliniCARE-Bench, a clinician-calibrated audit benchmark built on MIMIC-IV, the best of 16 frontier agentic systems resolved only about 76% of cases correctly, and all of them over-committed on cases whose correct verdict was "indeterminate." We propose borrowing the Delphi method, an anonymous, multi-round survey protocol that expert panels use to reach agreement when evidence is incomplete. We use it in two places. First, a real clinician Delphi panel fixes how costly each kind of wrong answer is from the clinician's point of view. Second, a machine Delphi panel of language-model agents produces the verdicts. Each agent sees a different slice of the record, so disagreement that persists across rounds can point to the slice that is missing.
The opportunity is this. Fragmentation is usually treated as a data-cleaning problem. Delphi treats incomplete evidence as the normal case and makes disagreement visible instead of averaging it away. Subjective quality of service is the literature on why raw technical metrics diverge from what users actually value. It tells us how to score the result: by what a clinician would pay for each verdict, not by raw accuracy.
Core Claim
When agents are split by evidence source (notes, labs, medications, diagnosis codes, time windows), kept anonymous, and given only summary statistics and anonymized rationales between a small number of rounds, two things follow. The pattern of who stays in dissent separates "lack of data" from "medically ambiguous" cases better than the spread of votes among agents that all read the full record. When the decision threshold is set by a clinician-elicited cost matrix, the panel delivers higher clinician-weighted utility than voting, single-agent calibration, or judge-moderated debate at equal compute.
The mechanism is a correlated-error argument. Independent samples over the same full record share the same blind spots. When a key lab was done elsewhere, every sample can confidently infer "no abnormality" from its absence, so vote spread stays low exactly when it should be high. Agents restricted to different sources disagree in a structured way. The lab-only agent reports "no lab found," while the notes agent cites a referral. That disagreement survives controlled feedback because it reflects genuinely missing evidence rather than noise.
Prior Work and Where This Sits
Three strands motivate the proposal.
CliniCARE-Bench supplies the evidence for the gap. It documents under-abstention, and it shows that roughly one in five correct verdicts from the most affected systems relied on prohibited shortcuts such as diagnosing from billing codes. Its "Clinical Board" calibrated the benchmark, but the agents themselves never deliberate.
Nonstationary Gaussian Process Regression for Evaluating Repeated Clinical Laboratory Tests supplies the failure mode we simulate: a test that looks missing in local records may simply have been done at an outside center.
Act or Defer (STRATCP) wraps one fixed EHR model with class-stratified conformal action and deferral. Its "utility graphs" keep deferred prediction sets clinically coherent. STRATCP is the closest existing tool for deciding when to defer. It cannot tell why a case is uncertain, and it needs labeled calibration data for every class. We add a reason for deferral, and STRATCP-style thresholding remains a baseline.
Two papers on multi-agent debate both supply method and constrain it.
Voting or Consensus? introduces All-Agents Drafting, in which each agent writes a blind first answer before seeing peers, the equivalent of Delphi round one. It finds that consensus acts as a filter on knowledge tasks (54.0% vs 51.7% on MMLU) and identifies unanswerable SQuAD 2.0 questions. It also finds that voting wins on reasoning (54.8% vs 28.4% on MuSR), and that multi-round debate drifts. We reuse blind drafting, but we cannot assume consensus helps on reasoning-heavy audits; this is a real risk.
Peacemaker or Troublemaker shows that sycophancy tracks the abandonment of correct answers (r = 0.902). It reports that a centralized judge cuts disagreement collapse on CommonsenseQA from about 81% to 41%, and that debates degrade after 2–3 rounds. This constrains our design. We cap the protocol at three rounds, and agents never see each other's identities or full arguments, only medians, spreads, and anonymized rationales. This is the isolation Delphi was built to provide.
Clinical multi-agent pipelines such as Structured Evidence Routing (router, predictor, reviewer) and MoMA (modality specialists feeding an aggregator) already split work across sources. Both are sequential pipelines with no anonymity, no feedback statistics, and no stopping rule. MoMA's authors call their agent interactions "relatively simple." Our contribution is not splitting by source alone. It is using the persistence of dissent among source-split agents as the abstention signal.
The human side draws on clinical Delphi practice. VICCCS used a predefined 67% agreement cut-off to harmonize fragmented vascular cognitive impairment criteria. Like most such studies, it ended in a static document rather than in anything a machine can use. We produce a machine-usable artifact instead: a cost matrix.
From the quality-of-service side, How to Maximize User Satisfaction Degree in Multi-service IP Networks ("Network Worth") replaces raw throughput with priority-weighted user utility. We make the same move for audit verdicts.
ChatEHR shows why this matters in deployment. Benchmarks failed to predict interactive quality, and summaries carried an estimated 0.73 hallucinations per generation.
Research Program
Stage 1: clinician cost elicitation. A panel of roughly 15–25 clinicians runs a three-round Delphi with the 67% agreement threshold used by VICCCS. Its output is a cost matrix $C[t,p]$: the clinician-perceived cost of issuing verdict $p$ when the correct verdict is $t$. Both range over CliniCARE-Bench's four verdicts (Yes, No, Indeterminate: Lack of Data, Indeterminate: Medically Ambiguous). The panel also rates a bonus for a lack-of-data flag that names the missing source. We expect this bonus to be large, on the quality-of-service intuition that an actionable "indeterminate" is worth far more than a bare one. A system's clinician-weighted utility is
$$U = -\frac{1}{N}\sum_{i=1}^{N} C[t_i, p_i] + \frac{\beta}{N}\sum_{i=1}^{N} \mathbb{1}[\text{flag}_i \text{ names the missing source}],$$
where $N$ is the number of cases, $t_i$ and $p_i$ are the true and predicted verdicts for case $i$, and $\beta$ is the panel-elicited bonus.
Stage 2: the machine Delphi panel. Each case's record is partitioned by source and time window. One agent is assigned per slice, plus one agent that sees the full record. In round one, each agent independently produces a verdict, a confidence, and a cited rationale.
Rounds two and three proceed as follows. A coordinator, which is code rather than a language model, returns the vote distribution and anonymized rationales. Agents in the minority must either revise or justify their position against their own slice. A stability statistic records which agents moved and why.
The final verdict is chosen to minimize expected cost under $C$, using the round-three distribution together with a dissent-persistence feature: whether the dissenting agents are the ones whose slice lacks the relevant evidence. If this rule is learned, it is fit on a held-out split of scenarios.
Stage 3: controlled fragmentation. We take CliniCARE-Bench cases with determinate verdicts and delete the critical evidence. The deleted evidence is either a source table or a time window, mimicking outside-facility care. This yields cases whose correct answer becomes "lack of data" by construction. Each scenario has only 30 cases, so these synthetic cases are needed to give enough indeterminate examples for statistical power.
Baselines
The naive reference is a single agent reading the full record with CliniCARE-Bench's own prompt. Its operating point shows how much any method gains over doing nothing. The strongest published reference point is CliniCARE-Bench's best system at about 76% raw accuracy, with defect-free accuracy 4.8–14.8 points lower depending on the system. The benchmark paper does not report per-verdict abstention rates in the material available to us; no reported result found on Lacuna for indeterminate-class recall on CliniCARE-Bench. We will therefore recompute all metrics ourselves for every baseline.
The strongest direct competitor is self-consistency voting: the same model sampled $k$ times on the full record, with abstention whenever the winning vote share falls below a threshold. This is the cheapest way to turn disagreement into abstention. Following Voting or Consensus?, it may also be the better aggregator for reasoning.
The second is calibrated single-agent deferral in the style of STRATCP. Confidence scores from the full-record agent are thresholded using conformal calibration on a held-out split, with the threshold tuned to the same cost matrix. Since STRATCP itself wraps a structured foundation model, we adapt only its thresholding idea.
The third is judge-moderated multi-agent debate with full records and identities visible, using the centralized-judge setup from Peacemaker or Troublemaker, capped at three rounds.
The key ablation keeps every Delphi ingredient except source splitting: anonymous rounds with statistical feedback, but every agent sees the full record. A second ablation keeps source splitting but drops the feedback rounds, which amounts to a MoMA-like aggregator.
Every system uses the same base models (Llama 3.3-70B and Qwen3-32B, both used in the cited debate work). Each gets the same token budget per case, with $k$ for voting set to match the Delphi panel's total calls. Every system's threshold is tuned on the same calibration split under the same cost matrix.
The proposal beats the baselines if it achieves higher $U$ than self-consistency at matched tokens, with a 95% case-level bootstrap interval excluding zero. It must also have higher recall on the "Lack of Data" class without lower defect-free accuracy on determinate cases.
Why not just…?
Why not just use self-consistency voting with an abstention threshold? This is the serious threat. If source splitting adds nothing, vote spread over full-record samples will flag missing data as well as our panel does, and the proposal should collapse into "use self-consistency with a cost-aware threshold." Our correlated-error argument predicts the opposite on the controlled-fragmentation cases. When the deleted lab leaves no trace, full-record samples should agree confidently on a wrong determinate verdict. The full-record Delphi ablation and the voting baseline at matched compute settle this directly. We will report the result whichever way it falls.
Why not just calibrate one model's confidence, as STRATCP does? Conformal thresholding controls how often a model errs. It cannot distinguish missing evidence from genuine medical ambiguity, and those two call for different clinician actions: request records versus convene colleagues. STRATCP also needs per-class calibration labels, which the 30-case scenarios cannot supply. The test is the confusion between the two indeterminate classes, and the naming bonus $\beta$, which calibration alone cannot earn.
Why not just run debate with a strong judge? The judge in Peacemaker or Troublemaker still leaves about 41% disagreement collapse on CommonsenseQA. In a missing-data case, collapse is exactly the failure that matters: the agent holding the gap gives in to confident peers. Delphi's anonymity and statistics-only feedback are designed to keep that agent's dissent alive. We will measure disagreement collapse rate on the fragmentation cases for both protocols.
Why not just retrieve the missing records? In deployment, retrieving the records is the right fix whenever it is possible. A system still has to recognize that something is missing and name what it is before anyone can retrieve it. The naming bonus measures exactly that.
Proposed Experiments and Decisive Checks
The main experiment runs all systems on the 750 CliniCARE-Bench cases plus the controlled-fragmentation set. It reports clinician-weighted utility, defect-free accuracy, per-verdict recall, and disagreement collapse rate.
The decisive check is the comparison of the source-split Delphi panel against the full-record Delphi ablation and self-consistency on the fragmentation cases. If the advantage disappears there, the core claim is false.
A secondary check tests transfer to prediction. On EHRSHOT incident-risk tasks, we mask outside-encounter windows and ask whether panel dissent predicts which patients have masked history. Structured Evidence Routing reports AUROC 0.88 against CLMBR's 0.82 on pancreatic cancer, which serves as the reference predictor.
Finally, a small clinician study tests whether the elicited cost matrix tracks clinicians' actual judgments. Clinicians rate blinded panel and baseline outputs on usefulness and time-to-resolution, and we check whether $U$ predicts these ratings better than accuracy does. If it does not, the quality-of-service framing has failed and only the abstention result stands.
References
- CliniCARE-Bench: Clinical Calibrated Audit of Medical Reasoning in EHR
- Nonstationary Gaussian Process Regression for Evaluating Repeated Clinical Laboratory Tests
- Act or Defer: Error-Controlled Decision Policies for EHR Foundation Models
- Voting or Consensus? Decision-Making in Multi-Agent Debate
- Peacemaker or Troublemaker: How Sycophancy Shapes Multi-Agent Debate
- Structured Evidence Routing for Incident Risk Prediction from Multimodal Longitudinal EHRs
- MoMA: a mixture-of-multimodal-agents architecture for enhancing clinical prediction modelling
- The Vascular Impairment of Cognition Classification Consensus Study
- How to Maximize User Satisfaction Degree in Multi-service IP Networks
- Adoption and Use of LLMs at an Academic Medical Center
- EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models