Can JEV Support Clinical Trial Criteria Assessment?

Mengxian He, Ziyue Gao, Chun Li, Shunyi Zhang, Hao Wei, Minqing Zhang, Wu YuanarXiv 2026 · arXiv

TL;DR

This paper evaluates JEV, a non-generative decision model, for clinical trial prescreening, comparing it against the generative Qwen Flash. While both models perform similarly on standard criteria, JEV significantly outperforms Qwen in identifying "insufficient information" in patient records—doing so at a fraction of the cost and latency.

What Problem Does This Paper Address?

Clinical trial prescreening is a high-stakes bottleneck in medical research. It requires matching complex patient records against strict inclusion and exclusion criteria. The challenge isn't just a binary "yes/no" match; models must distinguish between a condition being clearly unmet and the record simply lacking enough evidence to make a call.

While generative Large Language Models (LLMs) like GPT-4 or Qwen are often used for this, they can be overkill—slow, expensive, and prone to "guessing" when information is missing. Furthermore, clinical data requires strict privacy, necessitating Zero Data Retention (ZDR) policies. This paper investigates whether a specialized, non-generative approach (JEV) can provide a more efficient and reliable "first pass" for clinicians.

Methods

The researchers compared two primary model architectures:

  1. JEV (jev-1.14.0): A non-generative decision model that takes a text context and questions as input and returns probabilities for predefined answer options (e.g., MET, NOT_MET, INSUFFICIENT).
  2. Qwen Flash: A generative LLM constrained by a strict JSON schema to ensure it only outputs valid labels.

Both models were tested in two modes:

  • Single: Each criterion is processed in a separate request.
  • Batch: Multiple criteria for the same patient are processed together, sharing the context.

The evaluation focused on classification quality (Macro-F1), the ability to handle missing data, and operational metrics like latency and cost.

Figure 1: Proposed role of JEV in clinical trial prescreening. (a) Reviewers assess the patient record against individual inclusion and exclusion criteria and document their judgments. (b) JEV assesses criteria sharing the same patient context in a batch, returning provisional labels and native option probabilities. Clinicians verify these judgments against the patient record and follow up on potential information gaps before making a screening decision.

Datasets

The study used 2,048 test patient-condition pairs across three primary benchmarks:

  • TrialGPT: A dataset of 1,014 criteria derived from actual clinical trials and synthetic patient summaries.
  • MIMIC-III clinical-trial questions: 532 Boolean questions derived from real-world clinical notes.
  • MIMIC-IV apixaban questions: 1,589 conditions focused on specific clinical parameters like LVEF (Left Ventricular Ejection Fraction).

Key Insights and Ideas

  • The "Admitting Ignorance" Advantage: The most striking finding was JEV’s ability to flag INSUFFICIENT information. Generative models like Qwen often defaulted to a "NOT_MET" label when evidence was missing, whereas JEV correctly identified the information gap. On the MIMIC-III dataset, JEV’s recall for insufficient information was 56.3%, compared to a meager 9.9% for Qwen.
  • Efficiency of Non-Generative Models: Because JEV doesn't need to "think" or generate text token-by-token, it is inherently faster. The paper demonstrates that for classification tasks with fixed options, generative power can actually be a liability in terms of resource consumption.
  • The Power of Batching: Processing multiple criteria for one patient simultaneously (Batch mode) drastically reduced latency and cost without sacrificing accuracy. For JEV, batching reduced median job time from ~8 seconds to just 1.2 seconds on the TrialGPT set.

Key Results

  • Performance: JEV and Qwen achieved comparable Macro-F1 scores on TrialGPT (~0.65) and MIMIC-IV (~0.81). However, JEV dominated on MIMIC-III with a Macro-F1 of 0.768 vs. 0.534 for Qwen.
  • Latency: JEV’s median job latency (1.18–1.40s) was significantly lower than Qwen’s (2.80–8.90s).
  • Cost: JEV was 53%–74% cheaper per condition than Qwen Flash.
  • Error Analysis: Confusion matrices (Figure 6) revealed that while JEV is better at identifying missing data, it still struggles with complex temporal relations (e.g., did a symptom occur before or after admission?) and multi-component requirements.
Figure 5: Comparison of quality vs. cost and latency.

Limitations and Future Work

  • Black Box Architecture: The authors note that JEV is accessed via an API (TypeSafe), and its underlying architecture and training data were not accessible for review.
  • No Explanations: Unlike generative models that can provide a rationale for their decision, JEV only returns labels and probabilities. This makes human verification more labor-intensive as clinicians must find the evidence themselves.
  • Clinical Errors: JEV still makes mistakes in interpreting clinical nuances, such as resolving conflicting laboratory results over time. The authors emphasize that JEV should be used as an initial assessment tool to flag candidates and gaps, not as a final decision-maker.

Future work could involve hybrid systems that use JEV for rapid screening and generative models for providing explanations on flagged cases.

Summary updated

Abstract

Clinical trial prescreening requires distinguishing unmet conditions from insufficient evidence in patient records. Given patient records, criterion questions, and predefined answer options, we retrospectively compared JEV, a non-generative decision model returning native option probabilities, with schema-constrained Qwen Flash across 2,048 test patient-condition pairs from TrialGPT and question-answer datasets derived from MIMIC-III and MIMIC-IV, all evaluated under strict Zero Data Retention (ZDR) policies to ensure clinical data privacy. Both models assessed conditions individually or in batches sharing the same patient context, with performance measured against reference labels. In batch mode, JEV and Qwen achieved similar macro-F1 scores on TrialGPT (0.652 vs. 0.660) and MIMIC-IV (0.807 vs. 0.814), whereas JEV substantially outperformed Qwen on MIMIC-III (0.768 vs. 0.534; $Δ$ = +0.234, 95% CI: 0.187-0.281). This advantage largely reflected higher recall for conditions labeled as having insufficient information (56.3% vs. 9.9%). Across all three datasets, JEV also had lower observed median job latency (1.18-1.40 vs. 2.80-8.90 s) and 53%-74% lower estimated cost per condition. These findings position JEV as an initial assessment component that provides criterion-level judgments and flags potential information gaps for review. JEV's observed errors in interpreting clinical evidence underscore the need to verify its outputs against patient records before they inform eligibility decisions.
BibTeX
@misc{he2026can,
  author = {Mengxian He and Ziyue Gao and Chun Li and Shunyi Zhang and Hao Wei and Minqing Zhang and Wu Yuan},
  title = {{Can JEV Support Clinical Trial Criteria Assessment?}},
  year = {2026},
  eprint = {2610.35721},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35721},
}
Figure