Separating Proposal from Executable Checking in LLM Agents for Hyperparathyroidism Management

Patients on dialysis often develop secondary hyperparathyroidism (SHPT). Their failing kidneys cannot clear phosphate or activate vitamin D, so the parathyroid glands overproduce parathyroid hormone (PTH). Managing it is a balancing act among three coupled blood markers: calcium, phosphate and PTH. The drugs that push one marker toward target push the others away. Active vitamin D lowers PTH but raises calcium and phosphate, and that rise feeds vascular calcification (Trevasc-HDK; The Non-Traditional Cardiovascular Culprits in CKD). The risks are also non-linear and depend on PTH level. In the Japanese MBD-5D cohort, calcium shows a U-shaped mortality curve when PTH is high. Phosphate is far more dangerous at high PTH. Cinacalcet roughly halves mortality only when baseline PTH is very high (Calcium/phosphate mortality by PTH tier; PTH-dependence of cinacalcet). This is the kind of problem where a language model asked to "recommend therapy" will produce fluent advice that quietly breaks one of the coupled constraints.

The obvious response at this intersection is a panel of organ-specialist LLM agents (a mineral agent, a vascular agent, an anemia agent) debating under a judge. This proposal argues that the more useful decomposition is different. One LLM proposes a medication change. A separate, non-LLM program then predicts what that change does to all three markers and scores the result against PTH-stratified risk. Its structured verdict goes back to the proposer for revision. The scientific question is which ingredient actually reduces unsafe recommendations: role-play debate among agents, or an executable check. We answer it with a factorial ablation on real longitudinal hospital records.

Core Claim

In multi-target chronic disease management, recommendations go wrong mainly because the generator does not compute the downstream consequences of its own proposal on the other markers. Debate does not repair this. An executable "coupled-consequence checker" closes most of the gap, whether it is paired with a single agent or with many. Organ-specialist debate adds little beyond it.

The checker has two parts. The first predicts the direction and size of next-visit calcium, phosphate and PTH changes given a drug change. The second applies a risk layer encoded from published MBD-5D stratified estimates. If the claim is right, a single LLM with the checker should match a multi-agent panel with the checker. If it is wrong, debate should add a measurable reduction in coupled violations even when the checker is present.

The checker follows the "LLM minimization principle" of MALADE. MALADE is a pharmacovigilance system that hands frequency computation to code and keeps LLMs for interpreting text. We apply the same split to treatment planning, which MALADE does not attempt.

Research Program

Setting and position relative to prior work. Existing medical multi-agent systems motivate the work but do not test the hypothesis.

  • MedCoAct splits single-encounter consultations into a Doctor agent and a Pharmacist agent with confidence-triggered reflection.
  • AMR routes multiple-choice questions to specialist agents that each have their own memory.
  • MedOrch has a mediator model question open-source vision experts.

All three report gains over single models, but on static or one-step tasks. None separates the effect of a deterministic verifier from the effect of agent dialogue. MedOrch also constrains our design: it found that 7B models fail in the mediator role and need around 32B. Our judge and proposer therefore use the larger tier, and smaller models are tested only as proposers.

Clinical Decision Transformer supplies the sequential framing and the strongest non-LLM competitor. It is a goal-conditioned sequence model that recommends diabetes medications to steer HbA1c (glycated hemoglobin, a single blood-sugar marker). It steers one target, whereas SHPT has three coupled targets.

Cohort construction. The cohort comes from MIMIC-IV hospital records: patients with end-stage kidney disease on dialysis who have repeated calcium, phosphate and PTH measurements and prescriptions of vitamin D receptor activators, cinacalcet and phosphate binders. The usable cohort size is unknown until extraction, and MIMIC-IV measurements are hospital-centred and sparse compared with outpatient dialysis records.

Each decision point becomes a case: the lab history, current medications and free-text context up to that time, plus the next observed labs. If access to MBD-5D individual-level data can be negotiated, it becomes a replication cohort with dense dialysis-unit measurements.

The checker, part one: marker-response model. This component is a small, transparent regression of next-measurement changes in calcium, phosphate and PTH on drug-class changes and current levels. It is fitted on a training split with inverse-probability weighting for treatment selection, following the time-varying-confounding concern raised in the MBD-5D analyses.

The checker, part two: risk layer. This component turns predicted marker states into flags using the published stratified estimates:

  • calcium below 8.5 or at or above 9.5 mg/dL when PTH is at least 300 pg/mL (aIRRs 2.04 and 1.99);
  • phosphate at or above 7.0 mg/dL, weighted by PTH tier (aIRR 3.23 versus 1.58);
  • cinacalcet initiation when PTH is below 300 pg/mL, where no benefit was found;
  • the triple target ranges whose joint attainment carries a 16.8% population-attributable fraction, versus 8.8% for calcium and phosphate alone (target-achievement PAF study).

The four systems. We build four systems that share one base model, one retrieval corpus of guideline text, and one token budget per case. The single proposer and the organ-specialist panel are each run with and without the checker in the loop. In checker-equipped variants, the checker's structured output (predicted next state plus flags) goes back to the proposer or judge, which may revise up to a fixed number of rounds.

Baselines

Observed clinician practice. The naive reference point is what clinicians actually did, scored by the same checker and by realized next labs. For context, MBD-5D reports baseline attainment of 55.8% for calcium, 63.3% for phosphorus and 39.1% for PTH (PAF study). We expect lower figures in a hospitalized MIMIC-IV population, and we compute the MIMIC-IV figures ourselves.

Guideline rule engine. This is a deterministic decision tree over the published target ranges that titrates binders, vitamin D and cinacalcet. It is the strongest competitor on the safety metric because it cannot hallucinate. It is run on the same cases with no tuning beyond the guideline thresholds.

Multi-target Clinical Decision Transformer. This is a Clinical Decision Transformer extended to condition on all three marker goals and trained on the same training split. It is the strongest learned competitor for matching effective treatments. On its own diabetes cohort it reports 94.4% recall of effective treatments in complex patients versus 77.4% for behavior cloning. There is no reported result on Lacuna for any SHPT or CKD mineral-disorder recommendation task.

MedCoAct-style pair. This is a Doctor/Pharmacist pair with confidence-triggered reflection, re-implemented with our base model. It represents published multi-agent practice. It reports 67.58% Top-1 accuracy on DrugCareQA, about 7% above single-agent retrieval (MedCoAct). That benchmark does not transfer to our task.

Key ablation. The factorial removal of debate and of the checker isolates which ingredient matters.

Fairness and the bar for beating baselines. All LLM variants receive equal prompt-engineering effort, judged on a held-out development split. The proposal "beats" a baseline if it lowers the coupled-violation rate without reducing triple-target attainment among concordant patients, with confidence intervals excluding zero.

Why not just…?

Why not just use a single strong LLM with guidelines in context? This is half of our factorial, not a dismissal. AMR's gains over GPT-4 (93.2% versus 86.1% on MedQA) came on exam questions that require no consequence computation. Our hypothesis is that a single LLM with the checker is enough. If the single LLM without the checker already matches the checker variants on coupled violations, the checker is unnecessary and the paper reports that.

Why not just use the guideline rule engine? This is a real threat on safety, because our checker is itself rule-like. The engine cannot read free-text context such as a recent parathyroidectomy, calcium-containing binders already prescribed, or a missing PTH value. It also cannot apply PTH-stratified benefit when thresholds conflict. If the rule engine matches checker-equipped LLMs on both violation rate and nephrologist-rated appropriateness, the honest conclusion is that LLMs add nothing here. We report that conclusion if it holds.

Why not just train a multi-objective sequence model? A goal-conditioned transformer can learn marker coupling implicitly. It also imitates historical practice, including its unsafe patterns, and confounding biases it whenever treatment choice tracks severity. We measure its coupled-violation rate directly. If it is lowest, the right decomposition may be the transformer as proposer plus the checker, and we run that variant too.

Proposed Experiments

Primary check. The primary metric is the coupled-violation rate: the fraction of recommendations whose predicted next state crosses a PTH-stratified risk flag. To keep the checker from grading its own homework, the same flags are also computed on the realized next labs in cases where the recommendation matches what the clinician actually prescribed. Those realized-lab results are adjusted with inverse-probability-weighted marginal structural models.

The claim survives if the checker main effect is large and the debate effect, given the checker, is near zero. The claim fails if debate reduces violations even when the checker is present.

Stratified checks. These test whether systems respect the non-linear evidence:

  • the rate of cinacalcet initiation at PTH below 300 versus at or above 500 pg/mL;
  • vitamin D starts when phosphate is at or above 7.0 mg/dL.

Blinded nephrologist review. A blinded nephrologist panel rates a stratified sample of cases for appropriateness. This guards against systems that game the checker by recommending nothing.

Cost. We report tokens and wall-clock time per case, since the panel variants multiply compute.

Checker robustness. We perturb the response model's coefficients to see whether conclusions depend on its accuracy.