A systematic review pools the trials on a clinical question and hands clinicians one answer. Each of those trials, though, only enrolled patients who passed its eligibility rules: age limits, lab thresholds, excluded comorbidities. Nobody routinely checks how many real patients fall inside the union of those rules. We propose to measure this directly. The system takes the trials included in a published review, turns each trial's eligibility criteria into queries that can run against a hospital's historical records, and reports the fraction of the hospital's actual patients who would have qualified for at least one trial in the evidence base. It also reports which criteria exclude the most people, and whether the excluded patients are systematically sicker.
This matters because the evidence already suggests that coverage gaps change the answer. When TrialTranslator manually emulated 11 oncology trials in a nationwide record database, real-world high-risk patients had median survival 62% lower than the trial results and a 5.7-month smaller treatment benefit. That analysis took hand curation of single landmark trials. We ask the review-level question automatically: across everything a Cochrane review pooled, who in the hospital was never studied?
Core Claim
The unit of analysis should be a whole review's included trials, not a single trial, and eligibility should be scored with three possible outcomes. Each real patient is eligible, ineligible, or undeterminable because the record lacks the needed fact. The last category matters because missingness in health records is not random: tests are ordered selectively for sicker patients, as documented by Ke Meng et al.. We therefore report coverage as an interval, not a point. Its lower bound counts only patients confirmed eligible for some trial. Its upper bound also counts patients whose missing facts could make them eligible.
The claim is testable in three parts. First, review-level coverage can be computed with per-criterion accuracy close to hand-built rule systems. Second, it differs from what a comparison of trial summary statistics would suggest. Third, uncovered patients concentrate in high predicted-risk strata, which would reproduce the TrialTranslator pattern without manual curation.
Relation to Prior Work
Two lines of work supply the machinery. CSMeD harmonizes 325 systematic reviews with their Cochrane protocols and included/excluded study labels. TrialPanorama links about 1.66 million trial records, including ClinicalTrials.gov eligibility text, to standard ontologies and trains 8B models for eligibility-criteria tasks. Both treat criteria purely as text evaluated against papers. We reuse them to obtain, for each review, its included trials and their criteria, and then run those criteria against patients.
The closest method is Lee et al., which normalizes ClinicalTrials.gov criteria into ICD, LOINC and CPT codes plus custom knowledge bases and executes SQL on a hospital warehouse at an F1 of 0.94. Their aim is forward recruitment for one trial at a time. They produce no review-level union and no treatment of undeterminable criteria. Their finding that standard vocabularies cover only 47% of attributes is a constraint on our design: code lookup alone will fail.
TrialTranslator motivates the question but differs in three ways. Its trials were hand-picked, its criteria were hand-translated, and its focus was survival emulation rather than coverage. Cohort design and NLP to reduce bias in EHR research shows that note-derived vitals cut missingness by 31% with Pearson r of 0.95–0.99. It motivates our use of clinical notes to shrink the undeterminable fraction. It also warns that how a cohort is sampled changes incidence estimates, so we fix and report the sampling frame.
Chester-Jones et al. show why the question needs asking. Their manual audit of early warning scores found 95% of development studies at high risk of bias, and it examined internal validity only. Coverage is the complementary question about external validity.
Research Program
Review selection comes first. From CSMeD we choose Cochrane reviews whose condition is common in MIMIC-IV, the public hospital and ICU record database from Beth Israel Deaconess. Candidate conditions include sepsis, acute kidney injury, heart failure and chronic kidney disease. We keep reviews whose included trials can be linked to registry eligibility text through TrialPanorama, targeting roughly twenty reviews.
Each trial's criteria are split into atomic conditions, for example "eGFR below 30" or "no prior dialysis." Each condition is compiled into a query over MIMIC-IV structured tables, using the MIMIC-IV pipeline and PyHealth 2.0 for loading and code mapping. Where a condition has no structured source, a note-extraction query runs over discharge summaries.
Every compiled condition returns true, false, or unknown for each patient. A patient is eligible for a trial if all of that trial's conditions are true. The patient is ineligible if any condition is false, and otherwise undeterminable. Review coverage is then the eligible fraction of the patient population at the lower bound, and the eligible-or-undeterminable fraction at the upper bound.
The population itself is defined from the review's own question (condition, setting, age), with index time at admission. We also compute, for each criterion, the patients it alone excludes, which gives a ranked list of the rules that most narrow the evidence base.
For ground truth, two clinicians label a stratified sample of patient–criterion pairs as true, false, or not determinable from the record. We will aim for several thousand pairs across reviews. They also label the full per-trial eligibility of a smaller patient sample.
Baselines
The naive reference point is code-only compilation. Each criterion is mapped to ICD or LOINC codes by string matching; criteria that cannot be mapped count as satisfied, and missing values count as not meeting the criterion. This is what a quick retrospective query does. Given Lee et al.'s 47% normalization figure, we expect it to overestimate coverage badly. It runs on the same patients, criteria and labels.
The strongest existing method is a reimplementation of the Lee et al. pipeline: domain categorization, normalization to standard terminologies, and hand-written knowledge-base rules for drug classes and similar concepts, executed as SQL on MIMIC-IV. Their reported result is an average F1 of 0.94 across eight clinical domains, with Cohen's kappa 0.82 against expert normalization, on the Sema4 warehouse. That number comes from different data and different trials, so it is a target, not a directly comparable score. We rerun the method on our labeled pairs. To keep the comparison fair, we give the knowledge-base authoring the same number of clinician-hours we spend on validation prompts and rules for our system.
The learned competitor pairs TrialTranslator-style translation with a current model. The TrialPanorama 8B eligibility model structures the criteria, and the same model with retrieval over the MIMIC-IV schema generates queries. It gets the same prompts budget and development split as our system. On this task TrialPanorama reports only relative gains over Llama-3-70B and wins over GPT-4o and o3-mini. No reported result was found on Lacuna for criteria-to-EHR query accuracy on MIMIC-IV.
Beating these baselines means two things. Per-criterion three-way macro-F1 on held-out reviews must match or exceed the Lee reimplementation. And the coverage interval must contain the clinician-labeled coverage more often than any baseline's point estimate lands within five percentage points of it.
Two ablations isolate the key ingredients. The first forces two-valued output, treating unknown as false, and measures how far coverage and the high-risk finding move. The second removes the note-extraction queries, to quantify how much of the undeterminable band notes resolve.
Why not just…?
Why not just compare each trial's reported baseline table with the hospital's summary statistics? This is the cheapest route: it reads mean age and comorbidity rates from the published papers and needs no queries. It is a real threat if it ranks reviews by coverage the same way our method does. It cannot say which patients are uncovered or which criterion excludes them, but that only matters if it gets the ranking wrong. We will compute a summary-statistic distance for every review and report its rank correlation with measured coverage. If the correlation is high, the individual-level machinery earns its cost only through the per-criterion diagnosis, and we will say so.
Why not just prompt a large model to write SQL from the criteria? CSMeD shows zero-shot GPT-3.5 losing to a specialized long-context model on full-text screening. TrialPanorama shows domain-tuned 8B models beating much larger general models on eligibility tasks. Neither result tests query generation against real records, though, so this remains open. The learned baseline above tests it directly. If it ties our system, our contribution narrows to the three-valued scoring and the review-level analysis, which the ablation measures separately.
Why not just run a full target-trial emulation, as TrialTranslator did? Emulation answers a stronger question, whether the treatment effect holds, but it needs survival data and confounding control for each trial. It is also fragile under unmeasured confounding, as Wentao Zhou et al. stress for propensity matching. Coverage makes no causal claim and can be computed for every trial in a review. It works as a screen that tells emulation efforts where to look, not as a replacement for them.
Proposed Experiments and Decisive Checks
The first experiment measures compilation accuracy. It reports per-criterion three-way macro-F1 on held-out reviews, broken down by the domains Lee et al. found dominant: conditions, lab tests and prior therapies. It also reports calibration of the unknown label, meaning how often "unknown" pairs are ones clinicians also could not determine.
The second experiment measures coverage validity. For patients with full clinician labels, we report how often our interval contains the true coverage, and the interval's width with and without note extraction.
The third experiment is the decisive substantive check. We train a standard in-hospital mortality model on MIMIC-IV with PyHealth. For reference, the MIMIC-IV pipeline reports an AUROC of 0.87 for a hybrid LSTM on a CKD ICU cohort, and PyHealth 2.0 reports 0.7547 for Deepr with default settings. We then test whether confirmed-uncovered patients are over-represented in the top predicted-risk tertile relative to covered patients, using a permutation test within each review. Finding such over-representation would reproduce TrialTranslator's high-risk gap automatically across many reviews. Finding no difference would count against the claim that coverage gaps matter. We commit to reporting either outcome.
A final robustness check recomputes coverage under a longitudinal sampling frame in the spirit of C3PO: patients with repeat admissions only. This tests whether the conclusions depend on how the hospital population was sampled.
References
- Evaluating generalizability of oncology trial results to real-world patients using machine learning-based trial emulations
- Risk factors for steroid-refractory in immune checkpoint inhibitor-induced colitis: a retrospective cohort study
- CSMeD: Bridging the Dataset Gap in Automated Citation Screening for Systematic Literature Reviews
- Developing Large Language Models for Clinical Research Using One Million Clinical Trials
- Establishing the Automatic Identification of Clinical Trial Cohorts from Electronic Health Records by Matching Normalized Eligibility Criteria and Patient Clinical Characteristics
- Cohort design and natural language processing to reduce bias in electronic health records research
- Maternal early warning scores shown to be methodologically weak and at high risk of bias
- An Extensive Data Processing Pipeline for MIMIC-IV
- PyHealth 2.0: A Comprehensive Open-Source Toolkit for Accessible and Reproducible Clinical Deep Learning
- Laparoscopic versus open pancreaticoduodenectomy for pancreatic ductal adenocarcinoma: a propensity score matching analysis