TL;DR
This paper presents a baseline for the SemEval-2024 Task 2, focusing on determining the inference relationship between clinical trial report sections and statements. The authors explore prompt optimization techniques for Large Language Models (LLMs), finding that Chain-of-Thought (CoT) prompting significantly improves performance over manually crafted prompts, especially in the challenging medical domain.
What Problem Does This Paper Address?
The paper tackles the critical challenge of ensuring the reliability and accuracy of Natural Language Inference (NLI) systems, particularly Large Language Models (LLMs), within the sensitive medical domain. While LLMs have shown impressive capabilities, their trustworthiness is paramount when decisions impact human lives. Traditional fine-tuning methods for LLMs are resource-intensive and raise privacy concerns with sensitive medical data. This work aims to find efficient and effective ways to adapt LLMs for NLI tasks in clinical trial reports without extensive fine-tuning, addressing issues like shortcut learning and performance degradation due to data shifts.
Methods
The authors explore three prompt optimization techniques:
- OPRO (Optimization of Prompt Instructions): This iterative approach uses a subset of development data to generate the most effective prompt instructions by evaluating their performance on a small set of test samples. While effective, it's resource-intensive.
- Zero-shot Chain-of-Thought (CoT) Prompting: This method allows the LLM to generate a step-by-step reasoning process (chain of thought) to arrive at a conclusion, without prior examples for the specific task. It uses a task-agnostic meta-prompt.
- Dynamic One-shot Chain-of-Thought Prompting: This technique builds upon zero-shot CoT by selecting a single, semantically similar example from a training dataset to guide the LLM's reasoning process. This involves creating a vector database of statement-reasoning path embeddings from the training data and retrieving the most similar example for each test sample.
They also investigated reformulation methods like paraphrasing and rephrasing negative statements, but these did not yield significant improvements.
Datasets
The primary dataset used is the SemEval-2024 Task 2 dataset. This dataset consists of pairs of clinical trial report sections and statements, where the task is to determine if the statement logically follows from the report (entailment vs. contradiction). The dataset was designed to assess the reliability and consistency of NLI predictions by including paraphrased texts and minor alterations that change the inference relationship. The authors also used a development dataset of 200 samples for training and assessing prompt strategies.
Key Insights and Ideas
- Synthetic CoT Prompts Outperform Manual Ones: The most significant finding is that automatically generated Chain-of-Thought prompts, especially in a zero-shot setting, substantially enhance performance compared to manually crafted prompts. This highlights the power of guiding LLMs through explicit reasoning steps.
- In-Context Learning as a Viable Alternative: The exploration of in-context learning (ICL) strategies, like dynamic one-shot prompting, demonstrates its potential as an indirect method of fine-tuning, especially when direct fine-tuning is not feasible due to data sensitivity or resource constraints.
- Importance of Reasoning Path: The success of CoT prompts underscores that the how of reasoning is as crucial as the final answer in NLI tasks, particularly in complex domains like clinical trials.
- LLMs Struggle with Paragraph-Level Matching: The paper notes that LLMs can sometimes struggle to identify the specific paragraph within a clinical trial report section that is most relevant to a given statement, suggesting a need for more precise contextualization.
Key Results
- The team ranked sixth in faithfulness score and fell outside the top 10 for baseline F1 score (0.70) and consistency (0.70).
- Zero-shot CoT prompting achieved the best F1 score (0.70) among the tested strategies.
- Dynamic one-shot prompting achieved the highest Faithfulness (0.89) and Consistency (0.71) scores, though its F1 score was lower (0.60).
- Mixtral-8x7B-Instruct was found to offer the best quality-to-time ratio.
- The authors observed that handcrafted prompts were generally less effective than optimized prompts or meta-prompts.
Limitations and Future Work
- Time and Resource Constraints: Due to time limitations, the prompt strategies were primarily trained and assessed on the development dataset, and the execution of the entailment task on the test dataset required 20 hours for each strategy.
- JSON Formatting Issues: The zero-shot CoT prompt approach sometimes returned answers in a non-JSON format, requiring a fallback to prioritizing the "contradiction" label when the entailment label was unknown.
- Future Work Suggestions: The authors suggest that implementing preprocessing steps, such as enriching clinical trial sections with additional information or transforming negative statements into positive ones, could improve performance. They also note the potential benefit of explicitly adding cohort numbers to section subtitles to help LLMs better match paragraphs to cohorts. Further research into reformulation methods and more advanced prompt engineering techniques is also implied.
Summary updated