A Time-Locked Replay Testbed for Antibiotic Stewardship Agents

Doctors treating a serious bloodstream infection must choose an antibiotic before they know what the bacterium is resistant to. Laboratory susceptibility results take one to three days. Choosing too broad a drug feeds resistance, and choosing too narrow a drug can kill the patient. Language-model agents, meaning models that call tools, read records, and revise plans over several steps, are an obvious candidate for this job. There is currently no reproducible way to test them on it. Machine learning for stewardship scores a single prediction at the time of culture. Agent benchmarks score coding, web browsing, and customer-service dialogue. Neither tells us whether an agent improves treatment as information arrives, or whether its confident explanations hide skipped safety checks.

We propose a testbed that replays real hospital infection episodes on a locked clock. The agent can see only the data that existed at each decision hour: vitals, labs, notes, Gram stain, species identification. It must commit to a regimen at each step. The testbed does not need to simulate how a patient would have responded to a different drug, because the main score depends only on the isolate's final measured susceptibility, and that is a property of the bacterium regardless of what was prescribed. With this, we can rigorously answer one specific question: does a tool-using agent beat a strong classifier that is re-run at each step? Our second question is whether transcript-reading judges miss safety omissions that the environment's own action log catches.

Core Claim

Simulators of patient physiology are attractive but cannot be validated: no one knows how a past patient would have responded to a different antibiotic. We avoid this by restricting the scored outcomes to quantities that do not depend on the counterfactual. Let $r_t$ be the regimen chosen at decision time $t$. Let $a(r_t)\in\{0,1\}$ equal 1 if every isolate in the episode's final susceptibility report is susceptible to at least one drug in $r_t$. Let $b(r_t)$ be the regimen's position on a fixed ordinal spectrum ladder agreed in advance with infectious-disease physicians, running from narrow agents up to carbapenems and colistin. We score each method by its frontier

$$U_\lambda = \frac{1}{|T|}\sum_{t\in T}\big(a(r_t) - \lambda\, b(r_t)\big),$$

which is traced over trade-off weights $\lambda \ge 0$. Here $T$ is the set of decision points in an episode. Reporting the whole frontier, rather than one blended number, keeps the choice of how much to value resistance pressure versus coverage out of the benchmark designer's hands.

Mortality is deliberately not a metric, because it is counterfactual. Our claim is that this frontier is enough to separate methods, and that the separation is driven by two specific abilities. The first is revising therapy as species and susceptibility data arrive. The second is using unstructured notes. This is testable by ablation.

Research Program

The episodes come from MIMIC-IV. We use positive blood cultures with full susceptibility panels, joined to timestamped prescriptions, labs, vitals, and clinical notes. Cohort extraction and temporal binning reuse An Extensive Data Processing Pipeline for MIMIC-IV, which standardizes ICU and ward extraction. We use it so that our cohort is reproducible rather than another custom pipeline.

Each episode has four decision points: culture draw, culture positivity with Gram stain, species identification, and final susceptibility release. At each point the environment exposes a small set of query tools: prior microbiology with isolate dates, current and past antimicrobials, creatinine and other organ-function labs, allergies, notes written before the clock time, and a prescribe action.

Leakage is the main threat to validity, so the clock lock is enforced inside the tool layer rather than the prompt. Notes, prescriptions, and labs time-stamped after $t$ are invisible. Discharge summaries are excluded entirely. Clinicians' later prescriptions are hidden, because they encode what the clinicians eventually learned. We hold out the most recent admission years as the test split. MIMIC-IV notes sit behind credentialed access, so pretraining exposure is less likely than for public clinical question sets. ClinicalLab showed that memorization inflates scores on public medical QA, so we still report results separately on the temporal holdout.

Trajectories are logged in the Agent Data Protocol format from Agent Data Protocol, which defines a shared schema of actions and observations for agent logs. Each tool query becomes an API action and each record returned becomes a text observation. Any harness that already speaks that schema can then run on the testbed without bespoke glue. We supply the method here; we are not building a new schema.

Runs execute in a containerized sandbox in the style of OpenHands. Every configuration is run with five seeds. OAgents found that unstandardized agent evaluations show large run-to-run variance, which motivates reporting seed variance as a first-class result.

The second component is state-checked safety scoring. Talk, Evaluate, Diagnose scores multi-turn agents with subgoal grading notes judged by a language model. It acknowledges that this misses "silent failures," where the agent says it did something the environment never registered. In stewardship, the dangerous silent failures are concrete and checkable from the action log. We check three of them, each fixed before any model is run:

  • prescribing an aminoglycoside or colistin without a renal-function query;
  • prescribing at culture draw without querying prior isolates;
  • leaving the regimen unchanged within a fixed window after a susceptibility report shows the current drug is inactive or needlessly broad.

The Oxford study found that time since a prior resistant isolate was the strongest predictor of resistance, which is why the prior-isolate query is one of the checks. We compute each check two ways, from the log and from an LLM judge reading the transcript. We also flag cases where the agent's text claims a check it never executed.

The third component is a controlled architecture comparison, implemented as interchangeable operators in AGORA. That work found that plain chain-of-thought often matches or beats elaborate agent algorithms at lower cost. ClinicalLab instead found that specialty-routed multi-agent teams outperform single models on diagnosis. These two papers disagree, and our setting can test the disagreement directly. Holding the backbone and tools fixed, we compare a single tool-calling loop against a two-role team (an infectious-disease consultant and a pharmacist). We record tokens and wall-clock latency alongside the frontier.

Baselines

The naive reference points are what actually happened and what a fixed protocol would do. First, we score the clinicians' recorded regimen at each decision point with the same coverage and breadth functions. Second, we score a written local guideline that starts piperacillin-tazobactam, escalates to a carbapenem for prior ESBL colonization, and spares carbapenems in urinary-source sepsis with aminoglycoside combination. The sparing rule follows Cariou et al., who report Day-30 mortality of 7% for non-carbapenem β-lactams versus 15% for carbapenems in ESBL urinary septic shock. Any method that cannot beat the guideline's frontier is not worth deploying.

The strongest existing method is the XGBoost susceptibility model from Machine learning to predict antibiotic susceptibility in Enterobacterales bloodstream infections compared to clinician prescribing. On Oxford data it reports AUC 0.68–0.74 before species identification and 0.83 after. In simulated empiric therapy it raised active treatment from 70% (clinicians) to 79% and optimal narrow-spectrum treatment from 26% to 34%. Those figures are on a different hospital system, so we reimplement the model on our MIMIC-IV cohort with the same feature families, and re-fit and re-apply it at every decision point using only clock-visible data. This sequential version is the real competitor. It already captures most of the benefit of revision without any language model. Regimens come from its predicted susceptibilities through the same narrowest-active-drug rule, swept across $\lambda$.

We add a second strong discriminative baseline: a multimodal model that fuses structured data with note text, following Affordable and real-time antimicrobial resistance prediction from multimodal electronic health records. That paper reports AUROC 0.7565 for gentamicin resistance and roughly a 10% AUROC gain from adding notes. This baseline tests whether an agent's advantage, if any, is just reading notes.

Fairness is enforced by giving every method the same episodes, splits, and visible data per clock time. Hyperparameter search for the classifiers and prompt search for the agents receive matched budgets of validation-set evaluations. Agents are additionally reported at matched token cost.

Ablations isolate the proposal's key ingredients:

  • an agent frozen after its first decision, which removes revision;
  • an agent with the note tool removed;
  • an agent given the sequential classifier as a callable tool, which is plausibly the best system overall.

Beating the strongest baseline means one thing here. The agent's frontier must lie above the sequential XGBoost frontier across the clinically relevant $\lambda$ range, with 95% bootstrap intervals (over episodes and seeds) that do not overlap at the pre-registered $\lambda$. It must also have a log-verified silent-failure rate no worse than the guideline's.

Why not just…?

Why not just re-run XGBoost at each decision point and skip the agent? This is the most serious threat, and it may win. Tabular models are cheap, deterministic, and do not hallucinate. The Oxford results already show large gains over clinicians from a single prediction. We do not argue around it. The testbed's value does not depend on agents winning: it is the first place where the two can be compared on identical time-locked information. If the sequential classifier matches the agent, the frozen-agent and no-notes ablations will show why. The likely conclusion would then be that the right design is a classifier with a thin agent wrapper for record retrieval, which is the classifier-as-tool configuration we already run.

Why not just grade transcripts with an LLM judge, as existing agent suites do? Judges reading transcripts cannot see actions that never happened. TED's authors flag this as their main limitation. We measure the gap directly: the proportion of log-detected safety omissions that the transcript judge rates as acceptable. If that proportion is near zero, state checking is unnecessary overhead and we will say so. If it is substantial, transcript grading is unsafe for clinical agents.

Why not just extend ClinicalBench with antibiotic questions? ClinicalBench evaluates diagnosis and history-taking through scripted consultations. It has no clock, no staged laboratory results, and no executable prescribing. Adding questions would recreate the static single-turn format that cannot measure revision after a susceptibility report. Revision is exactly the behavior our ablations show matters or does not.

Why not just simulate patient physiology so mortality can be scored? A simulated mortality curve measures fidelity to the simulator's assumptions, and no retrospective data can validate it. Restricting scores to coverage and breadth, which are computable from the isolate's real susceptibility profile, costs us mortality as an outcome. In exchange, every number in the benchmark is checkable against recorded data.

Proposed Experiments

The primary experiment runs every baseline and agent configuration on the temporal holdout and plots coverage–breadth frontiers per decision point. We use several open and closed backbones, so that the Fleming-style dependence on a single proprietary model is avoided.

The decisive check has two possible outcomes. Either the agent dominates sequential XGBoost at the pre-registered $\lambda$, or it does not. In both cases the ablations attribute the difference to revision, notes, or the classifier tool. Secondary checks are:

  • the transcript-judge miss rate on log-verified silent failures;
  • seed-to-seed standard deviation of $U_\lambda$ for each configuration, testing whether rankings survive variance;
  • single-loop versus two-role team at matched tokens, to see whether the AGORA finding or the ClinicalLab finding holds in a time-critical setting.

A sanity check confirms the testbed is not leaky. A model given only data available at culture draw should not reach the post-species discrimination that the Oxford study reports (AUC 0.83). If it does, future information is escaping through the tools.