TL;DR
This paper tackles the challenge of ensuring large language models (LLMs) can accurately understand and reason about clinical trial data, specifically for Natural Language Inference (NLI) tasks. The UniBuc team achieved a respectable 14th place in the SemEval-2024 Task 2 by using the SOLAR-Instruct LLM with a novel approach of tailored summarization and prompting for different sections of clinical trial reports, without any fine-tuning.
What Problem Does This Paper Address?
Clinical trial reports (CTRs) are dense and complex documents containing crucial information about medical studies. Manually extracting and understanding this information is a monumental task, given the sheer volume of trials available. Natural Language Inference (NLI) offers a promising avenue for automating this process, enabling machines to determine the relationship (entailment, contradiction) between a statement and a given text. However, clinical NLI is particularly challenging due to the need for domain-specific knowledge, numerical reasoning, and robustness against subtle semantic changes. The SemEval-2024 Task 2, NLI4CT, specifically aims to evaluate LLMs' ability to perform safe and reliable NLI on clinical trial data, including interventions designed to test model robustness against semantic alterations.
Methods
The UniBuc team's core strategy revolved around leveraging the SOLAR-Instruct LLM in a zero-shot and few-shot setting, focusing on input manipulation rather than model fine-tuning. Their approach involved two key components:
- Targeted Summarization: They summarized each section of the Clinical Trial Reports (CTRs) to reduce the input length and focus the model on essential information. This was particularly effective for the "Results" section.
- Tailored Prompting: Prompts were carefully crafted and customized for each specific section of the CTR (eligibility criteria, intervention, results, adverse events) and for both single and comparison trial types. This allowed for more precise guidance to the LLM.
They also experimented with pre-trained masked language models (MLMs) like PubMedBERT, XLM-RoBERTa, and DeBERTa, as well as other LLMs like LLaMa-2 and Mistral, but found that the instruction-tuned SOLAR-Instruct model, without fine-tuning, yielded the best results.
Datasets
The primary dataset used was the NLI4CT dataset (Jullien et al., 2023a). This dataset is specifically designed for Natural Language Inference in the clinical domain and contains data from Clinical Trial Reports (CTRs). It includes 1,000 breast cancer CTRs with 24,000 entailment relations, annotated by clinical domain experts. The dataset is structured with four sections per CTR: eligibility criteria, intervention, results, and adverse events. The NLI4CT dataset is unique in that it includes semantic-preserving and semantic-altering interventions on hypotheses to test model robustness.
Key Insights and Ideas
- Instruction-tuned LLMs shine without fine-tuning: The SOLAR-Instruct model, designed for instruction following and with enhanced mathematical capabilities, performed remarkably well on this task without any task-specific fine-tuning. This suggests that for complex reasoning tasks, the quality of instruction tuning can be more critical than extensive domain-specific pre-training or fine-tuning.
- Summarization is key for context reduction: Summarizing CTR sections significantly improved performance, likely by reducing the input length and making the task more manageable for the LLM. This also drastically reduced inference time.
- Tailored prompts are crucial for section-specific reasoning: Customizing prompts for each CTR section allowed the model to focus on the relevant information and perform better than a generic prompt.
- Causal models outperform MLMs for this task: The authors were surprised to find that LLMs (causal models) significantly outperformed BERT-based masked language models, attributing this to the LLMs' superior reasoning capabilities.
- Models struggle with semantic-preserving changes: Despite good performance on semantic-altering interventions (like contradictions), the models showed a tendency to rely on heuristics and struggle with semantic-preserving changes, such as paraphrasing. This led to perfect recall scores in some cases, indicating oversensitivity to wording.
Key Results
The UniBuc team achieved a consistency score of 0.72, ranking 14th out of 32 participants on the SemEval-2024 Task 2 leaderboard. Their best-performing model, SOLAR 10B Instruct v1.0, achieved an F1-score of 0.71 in the control set and 0.72 in the contrast set. They noted that their approach would have placed them 5th in the previous year's task, highlighting the progress made.
Limitations and Future Work
The authors acknowledge several limitations:
- Information loss during summarization: Summarization, while beneficial for efficiency, carries the risk of discarding essential information.
- Heuristic reliance: The model exhibited a tendency to rely on simple heuristics, particularly when dealing with semantic-preserving changes, leading to potential over-sensitivity to wording.
- Numerical reasoning challenges: Despite SOLAR's mathematical tuning, the model struggled with variations in measurement units and domain-specific acronyms.
- Prompting strategy limitations: While few-shot prompting generally improved results, higher shot counts (e.g., 6-shot) led to performance degradation, possibly due to context window limitations or model "forgetting."
Future work directions include:
- Investigating more robust methods for evaluating LLM reliability.
- Exploring strategies to offload complex numerical reasoning to specialized models (e.g., xVal) to create more robust ensemble systems.
- Further research into optimizing summarization techniques and prompt engineering for improved performance and robustness.
Summary updated