TL;DR
TrialJev is an open-weight model designed for rapid patient-trial eligibility screening. By using a pointer head to score natural-language options in a single forward pass, it achieves accuracy comparable to generative LLMs while being nearly 3x faster and running entirely on local hardware, preserving patient privacy.
What Problem Does This Paper Address?
Matching patients to clinical trials is a high-stakes needle-in-a-haystack problem. A single patient might be a candidate for hundreds of trials, each with complex, nested eligibility criteria involving numeric thresholds, temporal constraints, and medical negations.
While Large Language Models (LLMs) have shown promise in this area, they face two major hurdles:
- Latency: Generative LLMs are slow because they decode verdicts token-by-token (autoregressive decoding). When screening thousands of patient-trial pairs, this cost becomes prohibitive.
- Privacy: Many high-performing models are behind proprietary APIs. Sending sensitive patient records to external servers often violates hospital data governance and privacy regulations.
The authors introduce TrialJev to bridge this gap, providing a fast, local, and accurate "first-pass" scorer to prioritize candidates for deeper clinical review.
Methods
TrialJev adapts the "typed-decision" architecture (previously seen in MedJev and kev) to the task of clinical trial eligibility.
- Architecture: It uses Qwen backbones (ranging from 0.8B to 27B parameters) adapted via Low-Rank Adaptation (LoRA).
- The Single-Pass Trick: Instead of generating text like "Eligible" or "Excluded," TrialJev uses a bilinear pointer head. This head looks at the model's hidden states at a specific "decision" token and at the boundaries of the answer options. It then calculates a normalized score for each option in a single forward pass.
- Inference Routes: The paper compares three distinct ways to get a verdict, as illustrated in Figure 1:
- (a) TrialJev: Scores option spans directly from hidden states.
- (b) A/B-logit Control: A classifier that looks only at the logits of tokens "A" and "B."
- (c) Generative LLM: The standard route of prefilling, decoding, and parsing a text verdict.
Datasets
The models were trained and evaluated using the TREC Clinical Trials benchmarks:
- Training/Development: TREC 2021 data, consisting of 75 synthetic patient cases and approximately 35,000 judgments.
- Evaluation: TREC 2022 data, featuring 50 unseen synthetic patients and 6,975 "eligible-or-excluded" pairs.
- Task Definition: The task is binary classification: determining if a patient is eligible or excluded for a trial they have already been deemed "relevant" to.
Key Insights and Ideas
- Decoding is the Bottleneck: The primary efficiency gain comes from avoiding autoregressive decoding. An ablation study (the A/B-logit control) showed that simply not writing text accounts for most of the speedup, rather than the specific pointer head architecture itself.
- Local Models Match Proprietary APIs: With proper calibration (fitting a temperature and class-prior weight on development data), the open-weight 27B TrialJev models achieved accuracy statistically indistinguishable from the proprietary, hosted Jev API.
- Initialization is Critical for Small Models: The smallest model (0.8B) failed to learn the task effectively from scratch. However, when initialized with weights from a related clinical extraction task (MedJev), its performance jumped by over 16 points.
- Calibration Nuances: While TrialJev is fast and accurate, the pointer head tends to be under-confident compared to the A/B-logit control. This highlights that a normalized score is not automatically a calibrated clinical probability and requires post-hoc adjustment.
Key Results
- Accuracy: The 27B models achieved 80.97% (Qwen3.5) and 81.84% (Qwen3.8) accuracy, outperforming generative LLMs fine-tuned on the same data (~79.9%).
- Speedup: In matched batch-1 measurements, TrialJev was 2.45x to 2.70x faster than the generative approach at the 9B–27B scale. At 27B, it required only ~178 ms per patient-trial pair.
- Latency vs. Accuracy: Figure 3 demonstrates the clear Pareto advantage of the single-pass approach, maintaining high accuracy while significantly reducing latency compared to generative and explanation-based baselines.
Limitations and Future Work
- Synthetic Data: The evaluation relied on synthetic TREC patients. Real-world clinical records are noisier and longer, which may impact performance.
- Binary Scope: The current model only distinguishes between "eligible" and "excluded" for relevant trials; it does not handle the initial retrieval of relevant trials from a registry.
- Lack of Explainability: Unlike generative models that can provide a rationale, TrialJev only provides a score. The authors suggest a staged workflow where TrialJev prioritizes candidates, and a slower, explanatory model reviews only the top-ranked pairs.
- Future Directions: The authors note that the pointer architecture could be extended to handle multiple questions (e.g., checking individual criteria) over a shared patient state, which could lead to even greater efficiency gains for long patient records.
Summary updated