TL;DR
This paper explores how well Masked Language Models (MLMs) and Large Language Models (LLMs) can determine if a statement is entailed or contradicted by clinical trial reports. They found that while LLMs with prompt engineering showed promise, a well-tuned MLM ensemble achieved comparable results with significantly lower computational cost, suggesting MLMs remain a strong contender for this task.
What Problem Does This Paper Address?
The paper tackles the challenge of Natural Language Inference (NLI) specifically within the domain of clinical trial reports (CTRs). Accurately understanding the relationship between a CTR and a given statement is crucial for tasks like patient recruitment and keeping researchers updated on medical literature. The NLI4CT task, introduced for SemEval-2024, focuses on evaluating the consistency and faithfulness of NLI models when applied to these complex medical documents, which often require nuanced reasoning beyond simple keyword matching.
Methods
The authors explored two primary approaches:
- Finetuning and Ensembling Masked Language Models (MLMs): They finetuned several MLMs (NLI-ROBERTa, ClinicalBERT, and Clinical-Longformer) on the NLI4CT dataset. To improve performance, they ensembled these finetuned models using both hard-voting (majority vote) and soft-voting (averaging probabilities).
- Prompting Generative Large Language Models (LLMs): They investigated using LLMs, specifically Flan-T5-large, with various prompt engineering techniques. These included:
- Simple Prompts: Basic instructions to perform NLI.
- Chain-Of-Thought (CoT): Decomposing the reasoning process by including evidence sentences in demonstrations.
- Contrastive Chain-Of-Thought (CCOT): Providing both correct and incorrect explanations in demonstrations to guide the model. They experimented with different few-shot settings (zero-shot, 1-shot, 2-shot) for these prompts.
Datasets
The primary dataset used is the NLI4CT corpus, which consists of breast cancer Clinical Trial Reports (CTRs) sourced from clinicaltrials.gov. This corpus was used for both training and development. The authors also mention using a "contrast set" of the gold test set for evaluating faithfulness and consistency metrics, which involves perturbed statements.
Key Insights and Ideas
- LLMs vs. MLMs Trade-off: The study highlights a common trade-off: LLMs with sophisticated prompting (like CoT and CCOT) can achieve good results without extensive finetuning, but they are computationally expensive. MLMs, while requiring finetuning, are more efficient.
- Prompt Engineering Matters: The effectiveness of LLMs was highly dependent on the prompt design. CoT and CCOT, while increasing prompt length, showed potential for improving reasoning, though they sometimes led to lower F1 scores due to increased complexity.
- MLM Ensembling is Robust: Ensembling multiple MLMs, even those with different strengths and weaknesses, provided a stable baseline and improved overall performance compared to single MLM models.
- ClinicalBERT's Anomaly: ClinicalBERT, despite being pre-trained on clinical data, showed a peculiar behavior of almost always predicting "Contradiction," leading to a 0.00 F1 score on the test set, though its faithfulness score was high due to the nature of the contrast set.
- Carbon Footprint Consideration: The paper explicitly discusses the significant carbon emissions associated with training and running LLMs compared to MLMs, making MLMs a more environmentally friendly choice for this task.
Key Results
The authors' best-performing system, using Flan-T5-large with a 2-shot prompt, achieved an F1 score of 0.57, Faithfulness of 0.64, and Consistency of 0.56. This placed them 27th out of 32 submissions for F1. Notably, an MLM ensemble system (combining finetuned NLI-ROBERTa, Clinical-Longformer, and ClinicalBERT, along with Flan-T5 predictions) achieved the exact same scores, underscoring the competitive performance of MLMs.
In terms of specific models:
- NLI-ROBERTa showed stable performance across metrics.
- Clinical-Longformer had the best F1 score among single MLMs but struggled with faithfulness and consistency.
- Flan-T5-large with 2-shot CoT or CCOT prompts showed improved faithfulness and consistency but sometimes lower F1 scores.
Limitations and Future Work
- LLM Computational Cost: The authors acknowledge the high computational cost and carbon emissions of LLMs, suggesting that for this specific task, MLMs might be more practical.
- Prompt Engineering Complexity: While effective, prompt engineering for LLMs can be complex and requires careful design.
- Future Work: The authors suggest continuing MLM pretraining on unlabeled clinical trial data. They also propose applying instruction-tuning to medical LLMs like MEDITRON and incorporating domain ontologies (like UMLS) into MLM finetuning to potentially improve performance.
Summary updated