Predicting a patient's health a year ahead from a decade of medical records forces a choice between two kinds of system. One is a single sequence model that reads the whole record cheaply. The other is a team of language-model agents that pulls out targeted evidence, reasons over it against clinical guidelines, and audits its own conclusions. The agent team is slower, but on some hard diagnoses it is clearly more accurate. This note proposes running the cheap model on every patient and calling in the agent team only for patients whose records the cheap model finds confusing. "Confusing" here means the model's prediction is uncertain, the record is unusually hard to predict, visits are irregularly spaced, or entries appear to be copied forward from earlier notes. When an escalated case still cannot be resolved from coded events alone, the agents may also consult a second channel, the clinician's free-text notes.
The underlying claim is empirical and can be falsified: the accuracy advantage of multi-agent decomposition on long-horizon prediction is concentrated in an identifiable minority of patients. If it is, a gate that routes only those patients recovers most of the accuracy gain at a small fraction of the cost. If the advantage is instead spread evenly across patients, gating cannot beat proportional savings, and the community should either pay for agents everywhere or not use them.
Why this question, and what prior work leaves open
Three findings in the literature point at the same gap. Context Clues trains long-context Transformer and state-space sequence models on structured records. It shows that per-token perplexity rises as patients age and that irregular visit spacing hurts every architecture. Its best model, Mamba-16k, reaches an average AUROC of 0.807 over 14 tasks of the EHRSHOT benchmark. This work motivates the proposal: it identifies a measurable property (rising perplexity and irregularity) that should mark the patients where a monolithic reader struggles. It never asks whether a different kind of system does better on exactly those patients.
Structured Evidence Routing (SER) supplies the method we reuse. It is a three-agent pipeline. A router reads the raw record and extracts compact evidence slices, a predictor applies guidelines to those slices, and a reviewer checks for temporal errors such as mistaking prior history for new onset. Its outputs feed a gradient-boosted readout. SER reports 1-year pancreatic cancer AUROC of 0.88 against CLMBR's 0.82, and lupus AUPRC of 0.18 against 0.14 for count-based models. It runs on every patient, and it reports no analysis of which patients account for its gain.
EHRSHOT constrains the design. It finds that count-feature gradient-boosted models catch up with foundation models once there are more than 64 labeled examples, particularly on distant-diagnosis tasks. Any claimed gain must therefore survive the cheapest serious baseline.
Adaptive routing exists elsewhere, but not in this form. AMR uses a moderator agent to send multiple-choice medical questions to low, moderate, or high complexity pathways. nMAS tiers oncology attribute extraction from regex up to a large model. In both, the routing decision is a judgment about the question's difficulty, not about how reliable a trained predictor is on a particular patient's history, and neither handles longitudinal prediction. AdaNav supplies the triggering idea: in vision-language navigation it invokes explicit reasoning only when action entropy spikes, adds under 0.4% latency, and raises success on R2R-CE val-unseen by 20%. We carry that idea from embodied steps to patients. The cheap model's uncertainty, combined with record-level difficulty signals, decides whether to escalate.
The interaction-design literature contributes the lens for the second channel. The "teletype" critique in Was the Title of This Talk Generated Automatically? and the fusion work in Goronzy et al. argue that forcing intent through one channel discards information that only another channel carries. Billing codes are such a single channel: a lossy summary of what a clinician observed. Multimodal medical agents such as MMedAgent-RL and 3MDBench handle images and dialogue, but only within a single encounter. The structured-record agents handle only codes. We test a narrow version of the multimodal claim: whether escalated agents gain from on-demand access to real notes specifically on the cases that coded data leaves uncertain.
The biomaterial literature, including Hsu et al. on 12-week magnesium anchor studies and Fujita et al. on bone barriers that resorb over decades, motivates the general concern: short evaluation windows miss late failure. It provides no benchmark, though; the goat study has two animals. We therefore do not claim results there.
Core Claim
On 1-year incident-diagnosis tasks from long structured records, a learned gate can identify a minority of patients who account for most of the gap between a strong sequence model and an always-on agent pipeline. Escalating only those patients matches always-on agent accuracy at a fraction of the language-model calls. For escalated patients, letting agents request free-text notes improves accuracy beyond codes alone, while adding notes for all patients through a simple embedding does not deliver the same gain per unit of cost.
Research Program
The system has three parts.
The first is the cheap model, a pretrained long-context sequence model in the style of Context Clues. It produces a risk score for every patient, along with an uncertainty estimate taken from a small ensemble's predictive spread.
The second is the gate. It receives that uncertainty plus four record-level signals:
- per-patient perplexity under the pretrained model, the quantity Context Clues showed rises with age;
- the variance of inter-visit gaps;
- a copy-forward proxy, namely the fraction of events exactly repeated from a previous encounter;
- record length.
The gate is a small classifier trained on the training split to predict whether the agent pipeline's prediction would differ from the cheap model's in the direction of the true label. Concretely, we run both systems on training patients, label each patient by whether the agent pipeline reduces that patient's log loss, and fit the gate to that label. A single threshold on the gate's output then sets the escalation budget.
The third part is the escalated pathway, a reimplementation of SER's router, predictor, and reviewer. In the multimodal arm, the router gets one additional tool that retrieves the patient's real notes within a requested time window. The reviewer is instructed to flag any claim that a note supports but the codes contradict, or the reverse.
The analysis runs before any method claim. We run the cheap model and the full agent pipeline on all test patients. We then stratify the per-patient improvement in log loss by quartiles of each gate signal. The claim survives only if the improvement is concentrated: for example, if the top quartile by perplexity or uncertainty carries well over a quarter of the total gain.
Baselines
The naive reference point is the cheap model alone with no escalation: the Mamba-16k-style sequence model trained on the same splits. The strongest reported number for this family is Context Clues' average AUROC of 0.807 across EHRSHOT tasks. Any configuration that spends language-model calls must beat it on the same tasks to justify the cost.
The count-feature gradient-boosted model from EHRSHOT is run with ontology expansion at every few-shot setting. It is the strongest cheap competitor when labels exceed 64, because EHRSHOT found that foundation models lose their edge there. For lupus, the file reports a count-model AUPRC of 0.14 via SER. For other tasks, we use our own reproduction.
The strongest existing method is always-on SER: the full router–predictor–reviewer pipeline run on every patient. Its reported figures are pancreatic cancer AUROC 0.88 versus CLMBR's 0.82, and lupus AUPRC 0.18, both from SER. The file also reports a "full framework" ablation figure of 0.86 AUROC for pancreatic cancer. Because the two SER figures differ, our reproduced always-on number is the operative target. All agent configurations share the same language-model backbone, prompts, guidelines, readout, and tuning budget. Only the escalation decision differs between them.
Two ablations isolate the gate. The first is random escalation at a matched budget. It controls for the possibility that any subset of agent calls buys proportional accuracy. The second is a set of single-signal gates, each using only one of predictive entropy, perplexity, or irregularity, which shows which signal carries the routing information. A further ablation, codes-only escalation, isolates the note-retrieval tool.
Beating the baselines means two things. At an escalation budget of about 20% of patients, the gated system should recover at least 80% of the always-on gain over the cheap model on AUROC and AUPRC. It should also strictly dominate random escalation across the budget curve, measured with paired bootstrap confidence intervals over patients.
Why not just…?
Why not just use a long-context sequence model and skip agents? It is the right default, and it is our cheap model. Context Clues itself shows that context scaling is fragile: Hyena collapsed at 16k tokens, and irregular visits hurt every model. SER's 0.88 against 0.82 indicates there is accuracy left on the table for at least some tasks. The stratified analysis tells us whether that remaining accuracy is reachable cheaply. If the agent gain is negligible on our reproduction, the proposal reduces to a negative result favoring the monolithic model, and we will report it that way.
Why not just use count features with gradient boosting? EHRSHOT shows this baseline is hard to beat at larger label budgets on distant diagnoses, so it is a real threat. We run it at every few-shot setting. We expect the gated system to win mainly at small label budgets and on multi-system conditions like lupus, where SER already beats counts. If counts match the gated system at high label budgets, we will say the gate's value is limited to the low-data regime.
Why not just run the agent pipeline on everyone? If cost is irrelevant, this may be optimal, and it is our upper reference. The multi-agent survey A survey on LLM-based multi-agent systems names inference overhead as a central obstacle, and SER's loop multiplies calls per patient. The budget curve settles the question directly. If accuracy rises linearly with the escalated fraction, gating offers nothing beyond what the budget would buy anyway, and always-on is the honest recommendation.
Why not just abstain on uncertain patients and send them to a clinician? Selective prediction is simpler and avoids language models entirely. It does not improve predictions on the abstained cases, though, and those are the ones we hypothesize agents fix. We will report the cheap model's accuracy on the retained set at each abstention rate next to the gated system's accuracy on all patients, so readers can judge the trade-off.
Why not just embed the notes and concatenate them into the cheap model? Early fusion is the obvious alternative to agent-requested notes, and it may capture most of the signal. We run it on the same cohort. The claim that escalated agents gain more from notes than blanket embedding does holds only if the codes-plus-notes agent arm beats the fused cheap model on the escalated subset.
Proposed Experiments
The primary experiments use EHRSHOT's 1-year incident tasks for pancreatic cancer, lupus, hypertension, hyperlipidemia, and acute myocardial infarction. They use the benchmark's official splits and few-shot settings, and report AUROC, AUPRC, Brier score, and language-model calls per patient. EHRSHOT has no notes or images, and the notes produced by MedNotes are generated from the same codes, so they add no new information. The notes arm therefore moves to MIMIC-III, which has real notes linked to coded data, using readmission among patients with multiple admissions as the task. No reported result was found on Lacuna for MIMIC-III readmission. Every number in that arm is therefore our own reproduction of all baselines.
Three checks decide the outcome. The first is the concentration test: if per-patient agent gains are flat across quartiles of every gate signal, the core claim is false and we stop at reporting that finding. The second is the budget-curve comparison: the learned gate must lie above random escalation, and at 20% escalation it must recover at least 80% of the always-on gain. The third is the channel test on MIMIC-III: on gate-selected patients, codes-plus-notes agents must beat codes-only agents, and must also beat early-fused note embeddings in the cheap model, at matched calls. If notes help equally on non-escalated patients, the multimodal benefit is real but has nothing to do with the uncertainty the gate detects, and we will report it that way.
References
- Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHR Data
- Structured Evidence Routing for Incident Risk Prediction from Multimodal Longitudinal EHRs
- EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Models
- Adaptive Memory and Reflection Multi-Agent System for Medical Question Answering
- From Analytics to Tumor Boards: An Evidence-Linked Multi-Agent Workflow for Oncology Feature Extraction
- AdaNav: Adaptive Reasoning with Uncertainty for Vision-Language Navigation
- Was the Title of This Talk Generated Automatically? Prospects on Intelligent
- Integrated development and on-the-fly simulation of multimodal dialogs
- MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoning
- 3MDBench: Medical Multimodal Multi-agent Dialogue Benchmark
- Fixation technique of biodegradable magnesium alloy suture anchor in the rotator cuff repair of the shoulder in a goat model: a technical note
- Surgical treatment for recurrent dislocation of the temporomandibular joint according to the Dautrey procedure.
- A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges
- A Multi-Agent Pipeline for Source-Grounded Synthetic Note Generation from Longitudinal Structured EHR