Hospitals increasingly expose patient records through FHIR (Fast Healthcare Interoperability Resources), a standard that represents each lab result, medication, or diagnosis as a small JSON document reachable through a web search interface. A natural way to answer a clinician's question over such records is to have a language model write a query or a short Python script, run it, look at the result, and fix the code if something went wrong. Privacy rules push hospitals to do this with small open-weight models running on their own hardware. This loop breaks in one specific place. When a FHIR query is wrong, the server usually does not raise an error. It returns an empty result. A misspelled lab code, an unindexed search parameter, and a patient who really never had the test can all produce the same empty bundle. A refinement loop that sees only this output has nothing to correct against, and a small model guesses.
We propose to turn empty results into diagnoses. The checker we would build is deterministic and costs no model calls. When a query comes back empty, it uses machine-readable artifacts the FHIR standard already requires servers to publish, together with a handful of cheap server probes, to work out why. It relaxes the query's constraints one at a time to find the one that emptied the result. It checks each search parameter against the server's declared capabilities. It looks up which codes the server actually holds for the resource type in question. It returns either a specific repair hint ("no Observation for this patient carries code X; the closest indexed codes are …") or a certificate that the empty answer is genuine. The claim is that this standard-derived feedback, rather than more model capacity or more training, is what lets an 8B model on one GPU close much of the gap to frontier cloud agents on FHIR question answering. We expect it to matter most on the two failure types the literature singles out: wrong clinical codes and genuinely empty answers.
Core Claim
Iterative code refinement works when the execution environment gives informative, external feedback. It degrades into cyclic edits or error amplification when the feedback is weak or comes from the model itself. FHIR is close to a worst case for weak feedback, because silent emptiness is the normal way a bad query fails. The same standard, though, carries the material needed to build a strong oracle. Our specific, testable claim has two parts. First, a checker that separates "query bug" from "true absence" will raise the accuracy of a local 8B agent on FHIR question answering, with gains concentrated in code-lookup and zero-result questions. Second, it will raise reliability, meaning correct answers on empty questions and fewer confident wrong answers. Both effects should hold more strongly than an equal-compute budget of extra sampling or plain error-message retry.
What Exists and Where This Sits
Three FHIR agent papers motivate the problem, and all three use frontier cloud models. FHIR-AgentBench builds 2,931 questions over MIMIC-IV-FHIR. It shows that letting agents write and run Python beats natural-language reasoning, and its best configuration reaches 50.0% accuracy. It also reports that 24% of questions have a genuinely empty answer, which is exactly where silent emptiness is ambiguous. FHIR-Hopper replaces free-form code with four deterministic graph-traversal tools. With Claude Sonnet 4.6 it reaches 0.611 on FHIR-AgentBench. However, it assumes a pre-built graph with intact references and reports 30–60 seconds per query. MedAgentBench provides a HAPI FHIR environment in which even Claude 3.5 Sonnet v2 falls from 100% on one-step tasks to 23.33% on tasks with three or more steps.
The work closest to ours is World Feedback for Clinical Agents. It trains a local Qwen3-8B on MedAgentBench-v3 using environment rewards. Pure reinforcement learning reaches 18.2% and rule-based supervised fine-tuning reaches 34.1%. The authors attribute the gap to "format-knowledge" failures: the model cannot find long SNOMED or NDC identifiers by trial and error. Their proposed fix is to put the knowledge into the weights, through a supervised warm start followed by reinforcement learning. We propose the opposite. We would supply the knowledge at inference time, from the server's own index, through feedback the agent can act on. The comparison between these two answers is a central experiment here, not a side note.
The method draws on local execution-feedback systems for SQL. MATS splits text-to-SQL into planner, validator, and fix agents totaling 9B parameters on one 24GB GPU, reaching 87.1% execution accuracy on Spider. Its validator flags "unexpected empty returns" but does not explain them. The AIRI EHRSQL system uses an execution-based inspector that catches empty or misleading results, but only converts them into abstentions and never repairs them. Our checker takes the step both stop short of: it localizes the cause of emptiness and decides whether emptiness is the right answer. The constraint-relaxation idea is in spirit close to the external oracles that made refinement work elsewhere. Examples are formal equivalence checking in Taneja et al. and rollback to the last working state in Fang et al..
Two papers act as warnings that shape the design. Yan et al. find that compiler-feedback gains plateau after one or two rounds, with models cycling through surface edits. Perry et al. show that iterating without a rigid outside check amplifies mistakes while user confidence stays high. Both imply that more refinement rounds are useless unless each round carries new information. Our checker is designed to supply that information.
Research Program
The first component is the checker. It runs beside a local HAPI FHIR server loaded with MIMIC-IV-FHIR for question answering, and with the MedAgentBench-v3 server for action tasks. It reads three kinds of server-published material. The first is the server's capability statement, which lists the search parameters it supports. Per the FHIR specification, servers may ignore unsupported parameters, and we will measure how often HAPI does so in practice. The second is resource profiles, used to validate any resource the agent intends to write before it is posted. The third is an inventory of the codes actually present per resource type, built by distinct-value queries over the loaded data. This inventory matters because real deployments use local, un-normalized codes, a divergence FHIR-AgentBench and FHIR-Hopper both flag. An external terminology server would miss these codes.
When an agent's query or script returns an empty result, the checker breaks the query into its constraints: patient, resource type, code, date window, status, and chained references. It drops them one at a time, and in pairs if needed, running each relaxed query against the server. This is a small delta-debugging search costing server calls rather than model calls. The minimal set of constraints that produces emptiness is reported back. If the culprit is the code, the report includes the nearest inventory codes by display-name similarity. If every constraint is individually satisfiable, the parameters are supported, and the codes exist in the inventory, the checker issues an "empty is genuine" certificate.
The second component is the agent loop. A Qwen3-8B model, quantized to fit a single 24GB GPU (the budget MATS shows is enough for a multi-agent SQL system), writes FHIR queries or Python over returned bundles. It receives checker reports as structured observations. Refinement stops on a non-empty result that passes the agent's own answer check, on a certified empty result, or after a fixed budget of rounds.
The third component is optional and kept separate so its effect can be measured. It is a small qLoRA fine-tune on the model's own successful repair trajectories, collected against the local server. No cloud teacher is involved, so the whole pipeline stays inside the hospital. Laboratory-Scale AI shows such fine-tunes are feasible on a single T4-class GPU.
For write tasks on MedAgentBench-v3, the checker runs profile validation and a reference-existence check before each create request, and refuses malformed writes. This targets the action-step collapse without needing rollback of records already committed.
Baselines
The naive reference point is the same quantized Qwen3-8B writing one query or script, executed once, with its output taken as the answer. A second reference is the trivial "always answer empty" policy. Reporting it on the zero-result subset shows how much of any gain comes from simply predicting emptiness more often.
The strongest local baseline is a MATS-style execution-feedback loop ported to FHIR, following MATS. It has planner, validator, and fix roles. The validator passes back raw server responses, Python tracebacks, and an "empty result" flag, and the fix agent retries. It is the strongest competitor because it is the best demonstrated small-model refinement design and differs from ours only in what the feedback says. It will use the same model, quantization, GPU, round budget, prompt-tuning budget, and maximum number of model calls. Our system must beat it on overall accuracy and on the reliability score defined below for the comparison to count as a win. Matching accuracy while needing more model calls would count as a loss.
The strongest reported local result on action tasks is rule-based supervised fine-tuning of Qwen3-8B at 34.1% on MedAgentBench-v3, from World Feedback for Clinical Agents. We will reproduce it, and also run its proposed supervised-then-reinforcement-learning warm start, on our hardware.
The strongest reported question-answering results are cloud-based. FHIR-Hopper with Sonnet 4.6 reaches 0.611 on FHIR-AgentBench (FHIR-Hopper), and the best FHIR-AgentBench configuration reaches 50.0% (FHIR-AgentBench). We treat these as ceilings to approach, not baselines we must beat. Exceeding 50.0% with a local 8B model would be a strong result. We have found no reported result on Lacuna for any model under 10B on FHIR-AgentBench, so our local numbers there would be new. We will also run FHIR-Hopper's four traversal tools with the local model, to test whether its tool design alone transfers to small models.
Two ablations isolate the key ingredient. The first gives the agent only a code-inventory lookup tool, with no constraint relaxation and no certificate. The second gives relaxation and certificates but no inventory hints. If the first matches the full system, our contribution reduces to "add a code search tool", and we will report it that way.
Why not just…?
Why not just do the supervised warm start that the World Feedback authors propose? It is a real competitor, and we run it. However, memorizing codes into weights addresses only one hospital's coding at training time. FHIR-AgentBench and FHIR-Hopper both document local code divergence, and frontier models still "scored 0% on some" format-knowledge tasks in World Feedback for Clinical Agents. That result suggests scale and training do not reliably supply such knowledge. To settle the question, we will train the warm start on one code distribution and test on held-out task types whose codes were never seen. If the warm start matches the checker on held-out codes, the inference-time argument fails.
Why not just sample more candidate queries and take a majority vote? Extra samples from the same model share its blind spot. If it does not know the right code, ten samples agree on wrong or empty answers, which is the cyclic behavior Yan et al. describe. We will give self-consistency the same number of model calls as our loop. Our prediction is that it gains on decision-type questions and not on code-lookup or zero-result questions.
Why not just use a frontier model under a business associate agreement? For institutions where that is allowed, it may simply be better. MedAgentBench-v3 frontier models reach about 78% (World Feedback for Clinical Agents). We do not claim otherwise. Our question is how much of that gap a local system can close, and whether the checker helps the frontier model too. We will test the latter on a small subset of the public MIMIC-IV-FHIR questions only.
Why not just give the small model FHIR-Hopper's traversal tools? Graph tools solve context overflow, but they still return nothing when a traversal filter is wrong. FHIR-Hopper also assumes intact references. Running its tools on the local model, with and without our checker attached, directly tests whether the two help in complementary ways.
Proposed Experiments
We will evaluate on FHIR-AgentBench, served from a local HAPI instance, using accuracy under its o4-mini grader. We will also use exact-match grading on questions with structured answers, so the on-premise result does not rest on a cloud judge. We will evaluate on MedAgentBench-v3 using pass@1 success, broken down by its decision, lookup, and format-knowledge categories. The main question-answering run uses one 24GB GPU, and a secondary run uses a 16GB T4.
Following the reliability framing of the EHRSQL 2024 shared task, we also report a penalized reliability score on FHIR-AgentBench. A correct answer earns credit, an incorrect answer costs ten times as much, and an abstention costs nothing. We report accuracy separately on the 24% zero-result subset. Latency is reported as wall-clock time and as model calls per question, against the 14–22 seconds MATS reports for SQL.
The decisive checks are four. First, before building anything, we will audit the naive and MATS-style loops' failures on a few hundred FHIR-AgentBench questions to measure what fraction are silent empty results. If this fraction is small, the premise is wrong and we stop. Second, we will measure the certificate's precision: when the checker certifies an empty result, how often the gold answer is truly empty. Below roughly 95%, the certificate is unsafe for clinical use. Third, the accuracy gain over the equal-compute MATS-style loop should fall mostly in code-lookup and zero-result questions, with little change on decision questions. A uniform gain would point to some other mechanism. Fourth, the code-inventory-only ablation and the held-out-codes comparison against supervised fine-tuning determine whether constraint relaxation adds anything beyond retrieval, and whether inference-time feedback beats trained-in knowledge when coding drifts.
References
- FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering
- FHIR-Hopper: A Neuro-Symbolic Agent for Reasoning over EHRs
- MedAgentBench: A Virtual EHR Environment to Benchmark Medical LLM Agents
- World Feedback for Clinical Agents: Diagnosing RL in FHIR Environments
- A Multi-agent Text2SQL Framework using Small Language Models and Execution Feedback
- AIRI NLP Team at EHRSQL 2024 Shared Task: T5 and Logistic Regression to the Rescue
- LLM-Vectorizer: LLM-Based Verified Loop Vectorizer
- MEIC: Re-thinking RTL Debug Automation using LLMs
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- Do Users Write More Insecure Code with AI Assistants?
- Laboratory-Scale AI: Open-Weight Models are Competitive with ChatGPT Even in Low-Resource Settings
- Overview of the EHRSQL 2024 Shared Task on Reliable Text-to-SQL Modeling on Electronic Health Records