Abstract
Clinical trials are critical for drug development. Constructing the appropriate eligibility criteria (i.e., the inclusion/exclusion criteria for patient recruitment) is essential for the trial's success. Proper design of clinical trial protocols should consider similar precedent trials and their eligibility criteria to ensure sufficient patient coverage. In this paper, we present a method named AutoTrial to aid the design of clinical eligibility criteria using language models. It allows (1) controllable generation under instructions via a hybrid of discrete and neural prompting, (2) scalable knowledge incorporation via in-context learning, and (3) explicit reasoning chains to provide rationales for understanding the outputs. Experiments on over 70K clinical trials verify that AutoTrial generates high-quality criteria texts that are fluent and coherent and with high accuracy in capturing the relevant clinical concepts to the target trial. It is noteworthy that our method, with a much smaller parameter size, gains around 60% winning rate against the GPT-3.5 baselines via human evaluations.
Introduction
Designing a clinical trial is a high-stakes, complex endeavor. One of the most critical components is the eligibility criteria—the specific rules defining who can and cannot participate. Errors in these criteria often lead to poor patient recruitment, safety risks, or inconclusive results. Currently, the process is manual and error-prone; approximately 57% of trial protocols require substantial amendments after the trial has started, with each amendment costing millions of dollars and causing significant delays. While general large language models (LLMs) are proficient at text generation, they often lack the domain-specific precision and reasoning required to navigate medical nuances without hallucinating or omitting key clinical concepts.
Methodology: The AutoTrial Framework
AutoTrial addresses these challenges using a decoder-based architecture enhanced by three core technical features: hybrid prompting, scalable knowledge expansion, and multi-stage training.
Hybrid Prompting
The framework utilizes a mix of discrete prompts (specialized tokens like <title>, <disease>, and <inc> to structure input) and neural prompts. These trainable embeddings allow the model to adapt to specific expert instructions without requiring full retraining, facilitating incremental learning.
Scalable Knowledge Expansion (RAG)
AutoTrial employs an "open-book" approach to knowledge. It uses Trial2Vec, a specialized encoder, to retrieve similar past trials from an external memory bank. These retrieved trials serve as "exemplars" to guide the generation process, ensuring the model leverages historical precedent.
Multi-stage Training
- Pretraining: The model is trained on a massive corpus of unlabeled trial documents to learn step-by-step reasoning and mimic the structure of successful trials.
- Finetuning: It is then tuned on a curated dataset of instruction-criteria pairs using a combination of Maximum Likelihood Estimation (MLE) and contrastive loss to ensure generated text is both fluent and distinct.
Experimental Setup and Datasets
The researchers curated a dataset of over 70,000 clinical trials from ClinicalTrials.gov. They applied a "Clinical Trial Parser" to extract over 750,000 medical relations from existing criteria to serve as ground truth. This resulted in approximately 2.5 million unique training samples for the pretraining phase.
Key Results and Evaluation
AutoTrial demonstrated a significant leap in performance over standard LLM baselines. It achieved a 0.91 F1-score and 0.84 Jaccard similarity in capturing relevant clinical concepts for inclusion criteria, whereas most baselines struggled to reach 0.5.
In head-to-head comparisons judged by domain experts, AutoTrial’s generated criteria were preferred over GPT-3.5’s outputs in approximately 60% of cases. Furthermore, it consistently outperformed other models (like GPT2-FT and T5-RAG) across standard NLP metrics including BLEU, METEOR, and ROUGE-L.
Discussion and Limitations
A key insight from this work is that a specialized model with 300M parameters can outperform a general-purpose model with 175B+ parameters when specifically tuned for the domain and equipped with a retrieval mechanism. However, the authors acknowledge several limitations:
- Data Dependency: The model's output quality is tied to the accuracy of the source clinical trial registries.
- Rare Events: The system may struggle with extremely rare side effects or unique patient circumstances not well-represented in the training data.
- Human-in-the-loop: AutoTrial is designed as a supportive tool for clinicians, not a replacement. Final safety and ethical evaluations must remain with human experts.
Future work will explore more complex aspects of trial design, such as defining outcome measures and optimizing recruitment strategies.
Summary updated