Knowing When to Stop Fixing: Noise-Robust Refinement for Clinical Text-to-SQL

Hospitals increasingly want language models to answer questions like "what was this patient's last potassium value before discharge?" by writing a database query, usually in SQL, against the electronic health record. Two facts from recent work pull against each other. First, a model that writes, runs, and repairs its own code in a loop usually beats one that writes code once. Second, in clinical querying a wrong answer is far worse than no answer. On the EHRSQL 2024 shared task, a system that refuses every question outscores several confident ones once errors are penalized. A repair loop is built to keep going until the code runs and returns something. That is exactly the pressure that turns a safe refusal into a confident, wrong clinical number. The problem gets worse once the question arrives inside a realistic, cluttered clinical context rather than as a clean sentence.

We propose a repair loop for clinical text-to-SQL whose stopping rule is a check against the database itself. For each question, the system writes one query for the question as it arrived, inside its noisy clinical context, and a second query for a version with that context stripped out. It runs both against the record database. If the two results agree, it answers. If they disagree, it repairs the queries using the concrete difference in returned rows as feedback. If they still disagree after a small number of rounds, it abstains. Every step is protected by rollback, so the loop can never replace an agreed answer or an abstention with something less consistent. We test this on a new noise-stress extension of EHRSQL. Clinicians review model-drafted distractors, and the gold query is re-executed to decide, deterministically, whether each distractor should change the answer at all.

Core Claim

Execution-feedback refinement, taken off the shelf into clinical settings, raises unpenalized accuracy but lowers penalized reliability. The cause is that it converts correct abstentions and noise-induced uncertainty into wrong queries. Agreement between queries written from the noisy and the stripped versions of the same request, checked by execution, is a better signal for both when to keep repairing and when to abstain. It beats the model's confidence, and it beats agreement across repeated samples of the same input. What is new here is using execution-level consistency across clinically meaningful rewrites as the stopping and abstention rule. Prior work uses execution only for pass/fail or for matching a synthetic reference.

Where This Sits in Prior Work

The clinical benchmark and its penalty come from EHRSQL and the EHRSQL 2024 shared task. EHRSQL built 24,411 question–query pairs over MIMIC-III and eICU from questions collected from hospital staff, and it handled abstention by thresholding output uncertainty. The shared task introduced the reliability score RS(c), in which each wrong answer costs c correct ones. We reuse both. What we change is that inputs arrive inside clinical context, and that the system is allowed to iterate.

The closest systems motivate the gap without filling it. KU-DMIS runs candidate queries against the database, but only to replace failing ones with an abstention, plus majority voting. Project PRIMUS shows what one-shot prompting does under penalties: RS(0) of 14.14 collapses to RS(10) of −713.37.

The refinement machinery is borrowed from general-domain work. OpenCodeInterpreter trains models to repair code from execution feedback. MEIC adds a scorer and rollback so that repairs cannot introduce regressions. TS-SQL catches silent semantic errors by running SQL against synthetic tables and comparing the result to reference Python code.

One paper also constrains the design. ErrorLLM reports that plain self-correction detects errors poorly, below 40 F1, and corrupts queries that were already correct. We take this negative result seriously, which is why our loop never repairs on the model's own say-so. It repairs only when two executions disagree.

The noise side is framed by three lines of work. LLMSenEval splits perturbations into those that should and should not change a diagnosis, and shows models failing both kinds on MedQA vignettes. Adaptive Distraction shows that random noise barely hurts models (under a 5% drop), whereas tree-searched single-sentence distractors cut accuracy by over 45%, and that "ignore irrelevant text" instructions do not help. Beyond MedQA argues that real decisions happen over redundant, noisy records.

We carry the should-change / should-not-change split of LLMSenEval into executable queries. The advantage is that the label comes from re-executing a minimally edited gold query rather than from physician judgment alone.

For data creation we follow the author-reviewer pattern. A model drafts and humans edit, which Building a Role Specified Open-Domain Dialogue System found roughly 13 times cheaper than writing from scratch.

Research Program

The benchmark: EHRSQL-Context. We take questions from the EHRSQL 2024 splits over MIMIC-IV Demo and wrap each one in a clinical context. The context takes the form of a clinician's request embedded in note-like text, containing non-contributory history, boilerplate, other patients' identifiers, unrelated lab panels, and competing time references.

Distractors come from two sources. One is an LLM drafter. The other is the tree search of Adaptive Distraction, adapted so that it maximizes disagreement with the gold execution result. Clinicians then filter drafts for plausibility and edit jargon back in. This targets the concern raised in EHRSQL that general paraphrasers smooth away clinical shorthand.

Each item receives one of three labels. An item is invariant if the gold result must not change. It is sensitive if the context legitimately changes a slot such as the time window or the lab name; in that case the gold SQL is edited and re-run. It is unanswerable if the context makes the request ill-posed, for instance by referring to data that are absent from the schema.

The labels are validated by execution. A candidate "invariant" distractor is rejected if a clinician's edit to the gold query changes its result. We record review time per item so that the synthesis cost itself is reported.

The method: differential-execution refinement. The system has three parts. An extractor rewrites the contextual request into a clean question. A generator writes a query for the full context and a query for the extracted question. An executor returns result sets for both.

When the two results agree, the system answers. When they disagree, the difference in rows is fed back to the generator. The prompt names which slots differ and which rows appear in only one result. As in MEIC, a scorer keeps the best-so-far state and rolls back any repair that reduces agreement. After k rounds, with k fixed on the development split, persistent disagreement triggers abstention.

We also fine-tune an open model on successful repair traces, in the style of OpenCodeInterpreter, so that we can separate the effect of the protocol from the effect of training.

Baselines

The naive reference points are two. The first is abstain-all, which the shared-task overview reports at RS(10) = 20.0. The second is one-shot generation with the same base model and prompting in the style of Project PRIMUS.

The strongest existing result on this data is the shared-task winner from LG AI Research and KAIST, at RS(10) = 81.32 on the EHRSQL 2024 test set; it used self-training with pseudo-labels. Its code and models are not assumed to be available, so it serves as a published target on the clean test split. We do not reimplement it.

The strongest competitor we can run matched against our method is a KU-DMIS-style pipeline: templatization, table selection, schema-guided self-reflection, majority voting, and execution-failure-to-abstention. For KU-DMIS the file gives only a relative gain of 26.5 RS(10) over its fine-tuned GPT baseline. We will report our reimplementation's absolute score on the clean test before comparing on noisy data.

We also transplant the strongest general text-to-SQL refiners onto the same data and base model. The first is ErrorLLM, which lifted GPT-4o on BIRD from 55.87% to 66.23% execution accuracy. The second is TS-SQL, which reports 70.93% on BIRD-dev. Both get EHRSQL's uncertainty-threshold abstention attached, with the threshold tuned on the same development split.

Fairness is enforced in three ways. All methods share one base model per comparison. All get the same cap on model calls and tokens per question. All use the same development data for every threshold and for k. This matters because our method spends two generations per round, so baselines get extra samples to equalize cost.

Beating a baseline means a higher RS(10) on the noisy test split, with a paired bootstrap interval excluding zero. It also requires answer coverage no lower than the baseline's at matched RS(0). This second condition rules out winning by refusing more.

Two ablations isolate the key ingredient. The first replaces cross-context agreement with agreement among repeated samples of the same noisy input. That is ordinary self-consistency at equal call budget, and it tests whether the rewrite matters or only the redundancy does. The second removes rollback, which tests MEIC's regression concern in this setting.

Why not just…?

Why not just run ErrorLLM or TS-SQL on EHRSQL? This is the most natural competitor, and we run it. Our concern is structural rather than about capability. Both methods are optimized to produce a repaired query, and neither has any notion of a context that should or should not change the answer. The shared-task results show that pushing toward always answering is punished under RS(10). We will report the "abstention conversion rate" for each system: the fraction of items where the one-shot model correctly abstained and the refiner then produced a wrong answer. If the transplanted refiners with tuned abstention match us on the noisy split, the contribution reduces to the benchmark, and we will say so.

Why not just strip the noise first and then run a standard pipeline? This is a real threat, because our extractor already does exactly this. Ext2Gen found that joint extraction inside generation beat separate compression modules. Less is More found that summarizing instructions can mistake background for requirements. Both findings suggest that extraction alone will sometimes delete a sensitive detail, such as a changed time window. The extract-then-generate baseline is our method with the full-context branch removed. If it matches us on sensitive items as well as invariant ones, then noise stripping suffices and the cross-check adds nothing.

Why not just use self-consistency voting or entropy thresholds for abstention? KU-DMIS voting and EHRSQL entropy thresholds are cheap and already work on clean inputs. However, repeated samples from the same noisy input share the same distraction. Adaptive Distraction's finding that models are most fooled by perturbations resembling their own reasoning suggests they will agree on the same wrong answer. The same-input ablation at equal budget settles this directly.

Why not just feed the whole record to a long-context model and skip SQL? UDA reports GPT-4 falling from 45.9 with retrieval to 0.4 with direct long-context input on financial tables. EHR aggregation over time-indexed tables is the same kind of numeric extraction task. We include one long-context run on a subsample as a sanity reference only.

Proposed Experiments

The first check tests whether the tension exists. On the clean EHRSQL 2024 test split, we compare one-shot generation with uncertainty abstention against the same model inside a plain OpenCodeInterpreter-style repair loop. The expected pattern is that RS(0) rises and RS(10) falls, and that the abstention conversion rate is non-trivial. If refinement raises RS(10) on clean data, the premise weakens, and the proposal shifts toward pure noise robustness.

The second check is the main comparison on EHRSQL-Context. Results are reported per label: invariant accuracy, sensitive accuracy, and correct abstention on unanswerable items. We also report RS(0), RS(5), and RS(10), plus the coverage–reliability curve traced out by varying k. The decisive outcome is our method beating the matched KU-DMIS reimplementation and the transplanted ErrorLLM/TS-SQL systems on RS(10) at equal coverage. The gain must appear on sensitive items too, not only invariant ones; that is the signature distinguishing it from noise stripping.

The third check asks whether tree-searched distractors are harder here than drafted ones. Adaptive Distraction's gap between static and adaptive noise predicts they should be. We also measure how many searched distractors clinicians reject as implausible, since a large rejection rate would mean the search mostly finds unrealistic attacks.

The fourth check covers transfer and cost. The benchmark construction is repeated on eICU to probe the single-site limitation of EHRSQL. We report clinician minutes per accepted item against a from-scratch authoring control on a small subset. This tells us whether the author-reviewer economics carry over to executable clinical items.