A molecular tumor board (MTB) is a hospital panel of oncologists, pathologists and bioinformaticians that reads a patient's tumor sequencing results and recommends a matched drug or clinical trial. Many of these recommendations are never carried out. One reason is that eligibility is checked by hand and late. The board has to confirm that the patient's lab values, prior drugs, dates and organ function meet criteria such as "absolute neutrophil count ≥ 1500/µL within 14 days before cycle 1." Language models are fairly good at reading such criteria. They are unreliable at the arithmetic and date reasoning the criteria demand. A natural fix is to have the model write a small program for each criterion and run it on the patient's structured record. That approach runs into a known weakness of execution-guided code repair: the run only reports crashes, and a program that runs cleanly but encodes the criterion wrongly is never flagged.
We propose to split the work between two agents that cannot see each other's reasoning. A code agent writes the eligibility program. A test agent sees only the criterion text and writes small synthetic patient records together with the answers it expects. The program is then refined against those tests, and edits are rolled back whenever they break previously passing cases. The claim is narrow. In clinical logic, where no test suite exists, a multi-agent split pays off because it produces an independent test oracle, not because it adds more "roles." That oracle should catch the silent logic errors that crash-driven refinement and self-written tests miss.
Core Claim
Iterative code refinement has worked where an external checker exists. MEIC repairs Verilog using compiler and simulator feedback plus rollback, and reports a 78% functional fix rate against 66% for one-shot retrieval-augmented baselines. MAGIS resolves GitHub issues against the repository's regression tests. CAAFE repairs generated feature code from Python tracebacks, and its downstream cross-validation score also acts as a checker. Clinical eligibility logic has none of these checkers. A traceback reveals a missing creatinine value. It does not reveal that the program compared against the wrong date window or used an inclusive bound where the protocol required a strict one.
Our claim is that the missing checker can be manufactured by a second agent kept deliberately isolated from the first. This isolation, rather than the multi-agent structure as such, should reduce the rate of "silent wrong" programs: programs that execute without error and still disagree with clinician adjudication. It should do so beyond what the same model achieves by writing its own tests, at matched compute. This is a specific instance of the confirmation-bias argument behind author and critic splits in Self-collaboration Code Generation via ChatGPT and Adversarial Review. Here it is made testable because the critic's output is itself executable.
Relation to Prior Work
The closest oncology systems either do not write code or do not decompose the task. The autonomous oncology agent has one GPT-4 orchestrator calling OncoKB, guideline retrieval, vision models and a fixed, pre-written RECIST calculator. It reaches 87.2% decision completeness against 30.3% for GPT-4 alone. It never synthesizes or debugs new logic. nMAS decomposes tumor-board preparation into tiered extraction agents. Its "validation" is verbatim string grounding, which checks that a value exists in the record and says nothing about whether a criterion computed from it is correct. AIVARI and VarLitAgent stop at variant interpretation and evidence curation. These papers motivate the proposal: they show tools and decomposition help in oncology, and they leave eligibility logic unexecuted and untested.
On the refinement side, MEIC supplies the rollback method and MAGIS supplies the developer and QA loop. Both assume the tests are given. Two further papers constrain the design. Gains from iterative refinement flatten after three to five rounds (Yan et al.), so we cap iterations and report the curve rather than promise open-ended improvement. SlopCodeBench shows code degrades under repeated specification changes. Trial protocols are amended often, so we test for this explicitly.
The application regime comes from Zero-Shot Clinical Trial Patient Matching with LLMs. That paper shows zero-shot GPT-4 is strong on the n2c2 2018 cohort-selection criteria and weaker on criteria that need temporal or numeric reasoning. It also supplies the strongest direct-judgment baseline. The MTB literature (Kuehn et al., Weiss et al.) documents the gap between recommendations and treatments actually given. It also reports that clinical deterioration during slow review is a major cause of that gap. Our contribution to the gap is therefore speed and earlier screening, not a claim that faulty eligibility checking is its main cause.
Research Program
We would build a sandboxed pipeline in which each eligibility or contraindication criterion becomes one Python function over a normalized patient table: labs with timestamps, medication exposures, diagnoses and performance status.
The code agent receives the criterion text and the table schema.
The test agent receives only the criterion text and the schema. It never sees the code, and it gets a separate context and, in one arm, a different base model. It writes between ten and thirty synthetic patient records with expected MET or NOT MET labels. It is instructed to target boundary values, missing data, unit conversions and date windows.
A refinement loop runs the program on the tests. It passes failures to the code agent as structured diffs of expected versus observed output. It also passes failures to the test agent, which may argue that a test is wrong but must state which phrase of the criterion justifies its label. A disputed test is dropped only if a third, adjudicating call sides with the code. Every accepted edit must keep all previously passing tests passing, otherwise it is rolled back, following MEIC. Iterations are capped at five.
The analysis has three parts. First, we would measure how often each arm yields silent-wrong programs. Second, we would measure how correlated the test agent's misreadings are with the code agent's, in same-model and cross-model pairings. Correlated misreadings are the mechanism by which independence could fail. Third, we would measure how programs behave when the criterion text is amended mid-stream.
Baselines
The naive reference point is direct judgment. A frontier model reads the patient's notes or table together with the criterion and outputs MET or NOT MET with a rationale, as in the zero-shot matching study. On n2c2 2018, that paper reports zero-shot GPT-4 beating the prior best system by 6 macro-F1 and 2 micro-F1 points. The file gives only the margin, not the absolute macro-F1, so we would re-run this baseline ourselves with the same model we use for our agents. It is the strongest direct competitor because it is cheap, already outperforms earlier systems, and needs no infrastructure.
The second baseline is one-shot code: the code agent writes the program once and we execute it, with no refinement. It isolates whether writing code helps at all over direct judgment.
The third baseline is crash-driven refinement in the style of CAAFE. The program is repaired only from tracebacks, for up to five rounds. It shows how much of the benefit comes from making the program run at all.
The fourth and most important baseline is a single agent with a code interpreter that writes its own tests and refines against them. It uses the same model, the same number of generated tokens and the same iteration cap. This is the ablation that isolates our key ingredient. If our system does not beat it, the independence claim fails.
We also include a sampling baseline at matched compute. Direct judgment is run with majority voting over as many samples as the token budget of the full pipeline allows, to rule out the explanation that our system is simply spending more compute.
All arms share the same base model, the same patient tables, the same prompts tuned on a held-out development split with an equal tuning budget, and the same per-criterion token cap. Beating the strongest competitor means a statistically significant reduction in silent-wrong rate and in false-eligible rate, the safety-critical error of declaring an ineligible patient eligible, over the self-testing single agent. This must hold on the held-out numeric and temporal criteria, with patient-level bootstrap confidence intervals, and without a loss in overall macro-F1.
Why not just…?
Why not just give one agent a code interpreter and ask it to test its own code? This is the real threat, and it is the fourth baseline above. The prior evidence against it is indirect. MALADE's critic corrected its primary agent in 44.5% of cases (MALADE), and author and critic splits help in code generation, which suggests that self-review misses errors. Neither result concerns clinical criteria. Our experiment settles the question directly. If self-written tests match independent tests at matched compute, we report that the isolation does not help, and the contribution reduces to showing that executable checks beat direct judgment.
Why not just prompt a stronger or reasoning-focused model directly? Direct judgment is our first baseline and is re-run with the strongest available model. The zero-shot matching study reports that it struggles specifically on temporal and numeric criteria. For those criteria we expect executed code to win on reliability, since arithmetic done by an interpreter is exact. We also expect code to win on auditability: a board member can read a ten-line function more easily than a free-text rationale. For simple criteria such as "diagnosis of X," direct judgment may well be as good or better. We will report results split by criterion type rather than hide this.
Why not just have clinical informaticians hand-code the rules? Hand-written rules are the gold standard for correctness. They are also what our adjudicated reference programs are, so they serve as the ceiling rather than a competitor. Their cost is authoring time for each trial and each amendment, and this matters most when referral is late, as the CUP cohort shows (CUP cohort). The comparison we would make is informatician time per criterion with and without our pipeline's drafts and tests, measured on a small user study.
Proposed Experiments
The first test bed is n2c2 2018. It has 288 patients and 13 criteria, and it is the only eligibility benchmark in our material with published LLM results. We would convert each patient's notes into our table schema using a fixed extraction step shared by all arms, and we would report results with and without gold-structured inputs so that extraction errors are not blamed on the reasoning stage. Its inter-annotator kappa of 0.54 caps the resolution any metric can have, so we treat small F1 differences cautiously.
The second test bed is a new oncology set, because no oncology eligibility benchmark appears on Lacuna. We would take numeric and temporal eligibility and contraindication criteria from published trial protocols, organized around targets discussed in MTB cohorts such as Trop-2-directed antibody-drug conjugates and kinase inhibitors. Clinicians would write reference programs for each criterion. We would then generate patient tables that include adversarial boundary cases and label them by running the reference programs. Because clinicians adjudicate the programs rather than label each patient, we can produce many boundary cases cheaply. The same arms are evaluated with a disjoint split of criteria reserved for testing.
The metrics are criterion-level macro-F1, the silent-wrong rate, the false-eligible rate, iterations until the pass rate stops improving, and tokens per criterion. The amendment experiment edits one clause in each criterion and measures whether updated programs regress on the unchanged clauses, with and without rollback.
The decisive check comes from comparing independent test authoring, from a same-model or different-model test agent, against self-authored tests at matched compute on the oncology held-out criteria. If the independent arm lowers silent-wrong and false-eligible rates, the proposal's mechanism is supported. If the gain appears only in the cross-model pairing, the lesson is that independence requires model diversity, and we would measure how often both agents make the same misreading of a criterion to explain why. If neither pairing beats self-testing, the multi-agent component is unnecessary for this task, and we would say so.
References
- MEIC: Re-thinking RTL Debug Automation using LLMs
- MAGIS: LLM-Based Multi-Agent Framework for GitHub Issue Resolution
- Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
- Self-Collaboration Code Generation via ChatGPT
- Adversarial Review: Structured Disagreement for Grounded Agentic Code Review
- Development and validation of an autonomous artificial intelligence agent for clinical decision-making in oncology
- From Analytics to Tumor Boards: An Evidence-Linked Multi-Agent Workflow for Oncology Feature Extraction
- AIVARI Agent: An Evidence-Grounded Agentic LLM for Variant Reportability and Interpretation
- VarLitBench and VarLitAgent for Benchmarking and Automating LLM-Assisted Functional Evidence Curation in Genomic Variant Interpretation
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- SlopCodeBench: Measuring Code Erosion Under Iterative Specification Refinement
- Zero-Shot Clinical Trial Patient Matching with LLMs
- Comprehensive genetic profiling and molecularly guided treatment for patients with primary CNS tumors
- Cancer of unknown primary (CUP) through the lens of precision oncology: a single institution perspective
- MALADE: Orchestration of LLM-powered Agents with Retrieval Augmented Generation for Pharmacovigilance