TL;DR
This paper explores the effectiveness of instruction-tuned Large Language Models (LLMs), specifically Flan-T5, for Natural Language Inference (NLI) on clinical trial data. By evaluating Flan-T5 in both zero-shot and fine-tuned settings, the authors demonstrate its strong performance, achieving a second-place F1 score of 0.834 on the NLI4CT task.
What Problem Does This Paper Address?
Interpreting and retrieving evidence from the ever-increasing volume of Clinical Trial Reports (CTRs) is a significant challenge for medical practitioners. Natural Language Inference (NLI) systems can help by determining the relationship (entailment or contradiction) between claims and the information within CTRs. However, applying NLI to the complex and specialized domain of clinical trials requires models that can effectively understand and reason over this data. This paper tackles the problem of building an effective NLI system for CTRs, specifically focusing on the Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) task.
Methods
The core of this work involves leveraging instruction-tuned Large Language Models (LLMs). The authors first evaluated several publicly available instruction-tuned LLMs in a zero-shot setting to gauge their baseline performance on the NLI4CT task. Based on these results, they selected the best-performing model, Flan-T5, and further fine-tuned it on the specific NLI4CT dataset. The input to the models was constructed using instruction templates, which included the CTR data (premise) and the annotated statement (hypothesis). For fine-tuning, they experimented with different instruction templates and optimizers, ultimately finding that a single instruction template and the Adam optimizer yielded the best results. They also employed advanced techniques like DeepSpeed ZeRO-3 for efficient training of large models.
Datasets
The primary dataset used in this research is the NLI4CT dataset, which is based on Clinical Trial Reports (CTRs) extracted from clinicaltrials.gov. The dataset consists of statements annotated by domain experts, focusing on four sections of CTRs: Eligibility criteria, Intervention, Results, and Adverse events. The task is divided into two subtasks: Task 1 for determining the entailment or contradiction between a claim and CTR evidence, and Task 2 for extracting the supporting evidence. The dataset includes both "single" CTR evidence and "comparison" CTR evidence scenarios.
Key Insights and Ideas
- Instruction Tuning is Key: The paper strongly suggests that instruction-tuned LLMs, like Flan-T5, are particularly well-suited for tasks with limited labeled data, such as the NLI4CT task. Their ability to follow instructions allows them to adapt effectively to new domains and tasks.
- Model Size Matters (for Fine-tuning): While zero-shot performance showed a general trend of improvement with model size, the fine-tuning process highlighted this even more. Larger models like Flan-T5-xxl showed a more significant improvement after fine-tuning compared to smaller variants.
- Prompt Engineering is Crucial: The authors observed that the choice of instruction template significantly impacts model performance, even for instruction-tuned models. This underscores the importance of careful prompt design.
- Fine-tuning Outperforms Zero-Shot: As expected, fine-tuning the Flan-T5 model on the specific NLI4CT dataset led to substantially better performance than zero-shot evaluation.
Key Results
- The fine-tuned Flan-T5-xxl model achieved an F1 Score of 0.834 on the official test set, securing the second position on the NLI4CT leaderboard.
- In the zero-shot setting, Flan-T5-xxl and Flan-T5-xl models demonstrated the best performance among the evaluated LLMs, with F1 scores of 0.727 and 0.703 respectively on the test set.
- The paper also noted that smaller Flan-T5 models, while performing poorly in zero-shot, showed significant improvement (up to +0.60 F1 score) after fine-tuning.
- Using a single instruction template for fine-tuning yielded better results than using multiple templates.
Limitations and Future Work
- Limited Model Scope: The study primarily focused on fine-tuning Flan-T5 models. Other models, like BioBERT, were only evaluated in a zero-shot setting, and their performance was noted as skewed due to their fixed positional embeddings not handling longer clinical text well. Non-generative models were skipped because they are not instruction-tuned.
- Task 1 Focus: The research concentrated on Task 1 (NLI classification) and did not explore combining Task 1 and Task 2 (evidence extraction) into a single model, as this would significantly increase input and output lengths, potentially impacting performance.
- Future Directions: The authors suggest exploring the combination of Tasks 1 and 2, and investigating the performance of other non-generative models if they can be effectively adapted. They also highlight the potential of instruction-tuned models for tasks with limited training data.
Summary updated