TL;DR
This paper evaluates JEV, a non-generative decision model, for clinical trial prescreening, comparing it against the generative Qwen Flash. While both models perform similarly on standard criteria, JEV significantly outperforms Qwen in identifying "insufficient information" in patient records—doing so at a fraction of the cost and latency.
What Problem Does This Paper Address?
Clinical trial prescreening is a high-stakes bottleneck in medical research. It requires matching complex patient records against strict inclusion and exclusion criteria. The challenge isn't just a binary "yes/no" match; models must distinguish between a condition being clearly unmet and the record simply lacking enough evidence to make a call.
While generative Large Language Models (LLMs) like GPT-4 or Qwen are often used for this, they can be overkill—slow, expensive, and prone to "guessing" when information is missing. Furthermore, clinical data requires strict privacy, necessitating Zero Data Retention (ZDR) policies. This paper investigates whether a specialized, non-generative approach (JEV) can provide a more efficient and reliable "first pass" for clinicians.
Methods
The researchers compared two primary model architectures:
- JEV (jev-1.14.0): A non-generative decision model that takes a text context and questions as input and returns probabilities for predefined answer options (e.g., MET, NOT_MET, INSUFFICIENT).
- Qwen Flash: A generative LLM constrained by a strict JSON schema to ensure it only outputs valid labels.
Both models were tested in two modes:
- Single: Each criterion is processed in a separate request.
- Batch: Multiple criteria for the same patient are processed together, sharing the context.
The evaluation focused on classification quality (Macro-F1), the ability to handle missing data, and operational metrics like latency and cost.
Datasets
The study used 2,048 test patient-condition pairs across three primary benchmarks:
- TrialGPT: A dataset of 1,014 criteria derived from actual clinical trials and synthetic patient summaries.
- MIMIC-III clinical-trial questions: 532 Boolean questions derived from real-world clinical notes.
- MIMIC-IV apixaban questions: 1,589 conditions focused on specific clinical parameters like LVEF (Left Ventricular Ejection Fraction).
Key Insights and Ideas
- The "Admitting Ignorance" Advantage: The most striking finding was JEV’s ability to flag INSUFFICIENT information. Generative models like Qwen often defaulted to a "NOT_MET" label when evidence was missing, whereas JEV correctly identified the information gap. On the MIMIC-III dataset, JEV’s recall for insufficient information was 56.3%, compared to a meager 9.9% for Qwen.
- Efficiency of Non-Generative Models: Because JEV doesn't need to "think" or generate text token-by-token, it is inherently faster. The paper demonstrates that for classification tasks with fixed options, generative power can actually be a liability in terms of resource consumption.
- The Power of Batching: Processing multiple criteria for one patient simultaneously (Batch mode) drastically reduced latency and cost without sacrificing accuracy. For JEV, batching reduced median job time from ~8 seconds to just 1.2 seconds on the TrialGPT set.
Key Results
- Performance: JEV and Qwen achieved comparable Macro-F1 scores on TrialGPT (~0.65) and MIMIC-IV (~0.81). However, JEV dominated on MIMIC-III with a Macro-F1 of 0.768 vs. 0.534 for Qwen.
- Latency: JEV’s median job latency (1.18–1.40s) was significantly lower than Qwen’s (2.80–8.90s).
- Cost: JEV was 53%–74% cheaper per condition than Qwen Flash.
- Error Analysis: Confusion matrices (Figure 6) revealed that while JEV is better at identifying missing data, it still struggles with complex temporal relations (e.g., did a symptom occur before or after admission?) and multi-component requirements.
Limitations and Future Work
- Black Box Architecture: The authors note that JEV is accessed via an API (TypeSafe), and its underlying architecture and training data were not accessible for review.
- No Explanations: Unlike generative models that can provide a rationale for their decision, JEV only returns labels and probabilities. This makes human verification more labor-intensive as clinicians must find the evidence themselves.
- Clinical Errors: JEV still makes mistakes in interpreting clinical nuances, such as resolving conflicting laboratory results over time. The authors emphasize that JEV should be used as an initial assessment tool to flag candidates and gaps, not as a final decision-maker.
Future work could involve hybrid systems that use JEV for rapid screening and generative models for providing explanations on flagged cases.
Summary updated