Language models answering questions about clinical documents often look accurate and fall apart under small, meaning-preserving edits. A model that says a trial "enrolled patients over 18" may give a different answer when "14 days" becomes "2 weeks", or keep its answer when a statement's negation is flipped. We propose a different setup. The model writes a small program that answers the question. Every number and phrase the program uses must be copied verbatim from the document. The program is then repaired in a loop driven by checks that need no human labels: does it run, does every value it uses actually appear in the source text, and does its answer change correctly when the question is perturbed in ways whose effect on the answer is known in advance.
The key idea is borrowed from software testing. Metamorphic tests are tests where you do not know the right output, but you do know how outputs must relate across transformed inputs. For example, inverting a negation must flip an entailment verdict, and rewriting a unit must leave it unchanged. Code-repair systems succeed when they have an external judge such as a compiler, a test bench, or a formal checker. Clinical text has no such judge, and this is the main reason iterative refinement has not reached it. We argue that metamorphic consistency combined with verbatim grounding can serve as that judge. We also argue that writing the reasoning as code is what makes failures under that judge repairable rather than merely detectable.
Core Claim
On clinical-trial natural language inference, a closed loop of program synthesis, execution, and repair will raise robustness to perturbations without sacrificing accuracy. The loop is guided by label-free metamorphic checks and a deterministic span-grounding check, and it rolls back any edit that breaks previously passing checks. Robustness here means the Faithfulness and Consistency metrics of the SemEval-2024 perturbation benchmark. The gain should exceed what the same backbone model gets from natural-language self-refinement under the same checks and the same call budget. The new element is not "LLM writes code for medicine." It is the source of the repair signal. That signal comes from known relations between perturbed inputs rather than from gold labels or a compiler, and it acts on an inspectable intermediate object, the program, so a failure can be traced to a specific extraction or comparison step.
Why This Gap Is Real
Three findings motivate the proposal. Jullien et al. ran the NLI4CT-P task, which tests inference over breast-cancer trial reports under paraphrase, contradiction, and numerical perturbations. They showed that high-F1 systems lean on surface shortcuts, and that very large models gave diminishing returns relative to mid-sized ones. Chen et al. showed on TheoremQA that offloading reasoning to executed Python (Program-of-Thought) beats natural-language chain-of-thought for GPT-4, reaching 51.0%. Their setting was one-shot generation with no repair loop and clean STEM problems rather than clinical documents. Das et al. showed that summarizing long trial reports before classification lifts Macro F1 but introduces semantic drift and hallucinated details. This is a direct argument for verbatim extraction over generative condensation.
Two findings constrain the design. Srivastava et al. found that loops relying on a model's own judgment amplify errors. Yan et al. found that feedback-driven refinement plateaus after a few rounds. Both imply that any loop needs an external, objective check and a small iteration budget.
Two code-repair systems supply the method. MEIC debugs Verilog with a scoring agent and rolls back edits that cause regressions. LLM-Vectorizer pairs generation with a formal equivalence checker. We reuse the rollback discipline and replace the formal checker with metamorphic and grounding checks.
Three systems are the closest relatives. CAAFE refines generated feature code using tracebacks and validation AUC, but on clean tabular data with gold labels available. AMR stores natural-language critiques of wrong answers on clean exam questions, and it needs to know an answer was wrong. LCDS applies medical logic to messy health records through static prompt templates and clinician review. None of the three obtains a repair signal at test time without labels.
Research Program
Each instance pairs a trial report with a statement. The model first produces an evidence table: a set of named fields, each holding a verbatim span from the report together with its character offsets. Examples are an arm's enrollment count, an adverse-event rate, or an eligibility criterion. Clinical noise enters at this step. Abbreviations, telegraphic phrasing, and dispersed evidence are handled by letting the model emit an explicit normalization function for each field, such as unit conversion or abbreviation expansion. That function is code and is tested like any other code. It is not a free-text rewrite.
The model then writes a short Python function over the table that returns entailment or contradiction. Execution yields the verdict.
Three checkers drive repair. The first is the interpreter. The second is a grounding checker: a string match that rejects any field value not found at its claimed offsets, and any program constant that does not come from the table or the statement. The third is a metamorphic checker. It perturbs the statement using generators whose effect on the label is fixed by construction: negation insertion or removal flips the label, unit and paraphrase rewrites preserve it, and a numeric shift across the program's own comparison threshold flips it. Where possible these generators are rule-based rather than LLM-based, so the oracle does not inherit model errors.
When a check fails, the failing evidence goes back to the model. That evidence is a traceback, an ungrounded span, or a perturbed statement with the program's wrong verdict. The model proposes an edit, and the edit is kept only if the number of passing checks does not drop. Otherwise the system reverts to the previous version, as in MEIC. The loop is capped at a small number of rounds, informed by the plateau result of Yan et al..
Our self-generated perturbations are built independently of the benchmark's evaluation perturbations and never consult gold labels, so the checks cannot leak the test set. We will additionally measure overlap in perturbation type and wording between our generators and the benchmark's perturbations, and report results separately by perturbation family.
Baselines
The naive reference point is the truncation baseline from Das et al.: DeBERTa-v3-base on truncated reports, which reports Macro F1 of 0.474 on NLI4CT. The summarize-then-classify pipeline from the same paper, which reports Macro F1 of 0.658, stands for the noise-filtering route. We will rerun both on the perturbation split so that Faithfulness and Consistency are available for them too.
The strongest existing reference is the best NLI4CT-P system reported by Jullien et al., Mixtral-8x7B-Instruct, with the highest F1 of 0.8. Jullien et al. report average Faithfulness of 0.835 and average Consistency of 0.751 across top models. These are pooled figures, not Mixtral's own. To keep the comparison about method rather than model size, every variant we build uses Mixtral-8x7B-Instruct as the backbone. We rerun Mixtral with direct prompting and with chain-of-thought under our own harness, so all reported numbers come from one evaluation pipeline. Prompt-tuning budgets are matched across methods on the development split.
The comparison that matters most is matched compute. Chain-of-thought with self-consistency voting, given the same number of model calls as our full loop, is the strongest budget-matched text-only competitor.
Ablations isolate each ingredient. One-shot Program-of-Thought in the style of Chen et al. uses the same programs with no repair. Traceback-only repair, in the style of CAAFE, removes the grounding and metamorphic checks. Further ablations remove grounding alone, metamorphic checks alone, and rollback alone.
The critical ablation is natural-language self-refinement driven by the same metamorphic checks. Here the model sees its inconsistent verdicts on perturbed statements and revises a chain-of-thought rather than a program.
We count a result as beating the strongest baseline if it matches Mixtral's F1 of 0.8 within noise and raises Consistency and Faithfulness above our rerun of Mixtral by a margin that holds across seeds. It must also beat the text-refinement ablation on Consistency at equal call budget.
Why not just…?
Why not just use a bigger model? Jullien et al. report that models beyond 70B parameters gave diminishing returns on exactly these robustness metrics and were often beaten by mid-sized models. Chen et al. find that scale does not substitute for missing domain distributions. Despite that evidence, we will include one larger model with direct prompting as a reference point. If it closes the Consistency gap, our approach is only worthwhile as a way to get comparable robustness from a smaller backbone, and we will say so.
Why not just run the same consistency checks on chain-of-thought answers, with no code? This is the real threat. If perturbation-driven text refinement matches the program loop, then the contribution is the metamorphic oracle, not the code. We expect code to win for two reasons. A failed numeric perturbation points at a specific comparison or unit-conversion line that can be edited. In text, there is no stable intermediate object to repair, so revisions tend to change the verdict without fixing the cause. In addition, verbatim grounding can only be checked deterministically when values are stored in fields. The text-refinement ablation decides this question directly. We will report its result whichever way it falls, broken down by perturbation family, because a win confined to numerical perturbations would be a narrower but still honest finding.
Why not just fine-tune an in-domain model or summarize first? In-domain encoders such as NYUTron are strong on raw notes. They are still limited by 512-token context, and they degraded when documentation style shifted during resident turnover. Summarization fixes length but, by its authors' own account, introduces semantic drift (Das et al.). Both routes are included as baselines and rerun on the perturbation metrics, so their robustness is measured rather than assumed.
Proposed Experiments
The primary evaluation is NLI4CT-P, scored with F1, Faithfulness, and Consistency. We report per perturbation family and plot score against refinement round to test for the expected early plateau.
A secondary evaluation is MedNLI (Romanov & Shivade), whose premises come from telegraphic MIMIC-III notes. It tests whether the grounding and normalization functions survive real clinical shorthand. We will focus on its negation and qualification cases, which Wang et al. identify as a persistent failure mode even for GPT-4. Wang et al. evaluate MedNLI but give no MedNLI score usable here, so the only reported reference is the 73.5% InferSent accuracy from Romanov & Shivade.
A noise stress test injects abbreviations and typographic variants into the trial reports and measures how much each method degrades. The injected patterns follow the abbreviation- and typo-driven vocabulary that the veterinary records study documents in clinical records. This test is synthetic, and we will present it as such.
Three checks decide the outcome. First, the full loop must beat text refinement with the same metamorphic checks on Consistency at equal call budget. Second, removing the grounding check must measurably raise the rate of answers built on values absent from the source report, as measured by our deterministic grounding checker. Third, gains on our self-generated perturbations must transfer to the benchmark's held-out perturbations. If the third check fails, the loop has learned to satisfy its own tests rather than to reason more robustly.
References
- SemEval-2024 Task 2: Safe Biomedical Natural Language Inference for Clinical Trials
- TheoremQA: A Theorem-driven Question Answering Dataset
- TLDR at SemEval-2024 Task 2: T5-generated clinical-Language summaries for DeBERTa Report Analysis
- Do Users Write More Insecure Code with AI Assistants?
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- LLM-Vectorizer: LLM-Based Verified Loop Vectorizer
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- Adaptive Memory and Reflection Multi-Agent System for Medical Question Answering
- LCDS: A Logic-Controlled Discharge Summary Generation System Supporting Source Attribution and Expert Review
- Improving Low-resource Question Answering by Augmenting Question Information
- Health system-scale language models are all-purpose prediction engines
- Lessons from Natural Language Inference in the Clinical Domain
- Are Large Language Models Ready for Healthcare? A Comparative Study on Clinical Language Understanding
- Text mining for disease surveillance in veterinary clinical data: part one, the language of veterinary clinical records and searching for words