TL;DR
This paper presents the NYCU-NLP system for the SemEval-2024 Task 2, which focuses on natural language inference (NLI) for clinical trials. By cleverly aggregating and fine-tuning large language models (LLMs), they achieved impressive results, ranking second in faithfulness and first in consistency among 32 participating teams. This work highlights the power of LLM aggregation and targeted fine-tuning for complex biomedical NLI tasks.
What Problem Does This Paper Address?
The core challenge addressed is Biomedical Natural Language Inference (NLI) for Clinical Trials, specifically in the context of evaluating the faithfulness and consistency of models when statements are manipulated. Clinical trial reports (CTRs) are complex documents, and determining whether a given statement is entailed, contradicted, or neutral by the CTR is crucial. The SemEval-2024 Task 2 introduces "interventions" – modifications to statements – to rigorously test how well models maintain their predictions when faced with subtle changes. This is vital for ensuring that AI systems used in healthcare can reliably interpret medical information, even when it's presented in slightly altered forms. Previous work in medical NLI often relied on BERT-based models, but the increasing capabilities of LLMs present new opportunities and challenges.
Methods
The NYCU-NLP system employs a multi-stage approach leveraging several LLMs:
- Data Augmentation (ChatGPT v3.5): To prepare for the intervention-heavy task, original statements were augmented using ChatGPT. This involved generating multiple rewritten versions of statements, with specific prompts designed to alter wording, sentence structure, and numerical representations (e.g., changing percentages to decimals). A special prompt was used to generate contradictory statements from entailment ones.
- Instruction Tuning (SOLAR v1.0): The augmented data was used to fine-tune the SOLAR model. This involved supervised fine-tuning (SFT) and direct preference optimization (DPO) with carefully crafted instructions to adapt the model for the NLI task. Techniques like LoRA and FlashAttention were used to optimize efficiency and performance.
- Intervention Reduction (OpenChat v3.5): To mitigate the impact of statement manipulations during testing, OpenChat was fine-tuned. This model acts as a "cleaner," aiming to identify and extract the core, unmanipulated meaning from intervened statements. This is achieved through few-shot prompt learning with examples demonstrating how to identify related sentences.
- Label Prediction (Fine-tuned SOLAR): The cleaned statements, along with the original CTRs, are then fed into the fine-tuned SOLAR model for the final label prediction (entailment or contradiction). The output is interpreted based on the first token of the model's response.
Datasets
The primary datasets used were provided by the task organizers for SemEval-2024 Task 2. This included:
- Clinical Trial Reports (CTRs): 1000 breast cancer CTRs served as the premise documents.
- Training Statements: 1700 statements were used for training, covering both single-CTR claims and comparisons between two CTRs, labeled as entailment or contradiction.
- Augmented Statements: Data augmentation generated 13,484 statements.
- Development and Test Sets: A total of 2,142 statements were used for system development and parameter optimization, and a final test set of 5,500 statements was used for evaluation.
The task itself is built upon the NLI4CT-2023 dataset, but with the crucial addition of manipulated statements designed to test faithfulness and consistency.
Key Insights and Ideas
- LLM Aggregation is Powerful: The success of the NYCU-NLP system demonstrates that combining and strategically fine-tuning multiple LLMs can yield superior performance compared to using a single model.
- Targeted Fine-tuning Matters: Simply using a large LLM isn't enough. The specific instruction tuning and intervention reduction steps were critical for adapting the models to the nuances of biomedical NLI and the challenges posed by manipulated statements.
- Faithfulness and Consistency are Distinct but Related: The results show that while models can achieve high faithfulness (correct predictions despite interventions), consistency (stable predictions for semantically similar inputs) is also a key metric. The system's top ranking in consistency is particularly noteworthy.
- Model Architecture Over Scale: The paper suggests that model architecture and fine-tuning strategies might be more important than sheer parameter count, as their SOLAR model outperformed larger models like Orca2 and Qwen on key metrics.
Key Results
The NYCU-NLP system achieved outstanding results in the SemEval-2024 Task 2:
- Faithfulness Score: 0.9236, ranking second out of 32 participating teams.
- Consistency Score: 0.8092, ranking first among all teams.
- The fine-tuned SOLAR model (10.7B parameters) outperformed Orca2 (13B), Qwen (14B), and Mistral (7B) on all tested metrics (F1, Faithfulness, Consistency) for both development and test sets.
Limitations and Future Work
- Limitations: The paper notes that the technical details of the interventions performed by the task organizers were not disclosed, which might have posed a challenge. Additionally, the process of cleaning intervened statements relies on the OpenChat model's ability to correctly identify and extract relevant information, which could be a source of error.
- Future Work: While not explicitly detailed, the success of this approach suggests future work could explore more sophisticated LLM aggregation techniques, novel methods for intervention robustness, and further investigation into the architectural factors that contribute to high faithfulness and consistency in biomedical NLI. The paper also hints at the potential for LLMs to outperform traditional models even without ensemble methods, opening avenues for more efficient and effective single-model solutions.
Summary updated