Lisbon Computational Linguists at SemEval-2024 Task 2: Using a Mistral-7B Model and Data Augmentation

Artur Guimarães, Bruno Martins, João MagalhãesNAACL 2024

TL;DR

This paper presents an approach to the SemEval-2024 NLI4CT task, which involves classifying entailment relations in clinical trial reports. By fine-tuning a quantized Mistral-7B model with augmented data, the authors achieved the top score on the leaderboard for classification accuracy, though with limitations in faithfulness and consistency.

What Problem Does This Paper Address?

The paper tackles the challenge of applying Natural Language Inference (NLI) models to specialized domains like medical data, specifically Clinical Trial Reports (CTRs). While Large Language Models (LLMs) excel at general NLI tasks, they often struggle with factual inconsistency, shortcut learning, and performance degradation when faced with domain-specific information. The SemEval-2024 Task 2 (NLI4CT) aimed to assess the accuracy, faithfulness, and consistency of LLMs in this medical context, requiring models to determine entailment or contradiction between statements and CTRs. This is crucial for ensuring reliable AI applications in critical fields like healthcare.

Methods

The authors employed a two-pronged approach:

  1. Model Choice and Fine-tuning: They selected Mistral-7B-Instruct-v0.2, a powerful open-source LLM, for its zero-shot capabilities and ability to handle long input texts (essential for lengthy CTRs). To optimize performance and efficiency, they quantized the model to 4-bit precision and used Low-Rank Adaptation (LoRA) for fine-tuning.
  2. Data Augmentation and Prompt Engineering: Recognizing the potential insufficiency of the original dataset for robust generalization, they augmented the training data. This involved creating new "Train_Manual," "Train_Manual-Synthetic," and "Train_Full-Synthetic" datasets by manually curating and synthetically generating entailment and contradiction examples. A significant effort was also dedicated to prompt engineering, where they systematically tested various prompt templates to find the one that yielded the best macro F1-score.

Datasets

The primary dataset used was the NLI4CT dataset for SemEval-2024 Task 2. This dataset consists of statements about Clinical Trial Reports (CTRs) and is divided into training, development, practice-test, and test splits. The task organizers specifically designed the dataset with controlled interventions (preserving or inverting entailment relations) to test model robustness. The authors also created augmented versions of the training set:

  • Train_Manual: Manually curated additional samples.
  • Train_Manual-Synthetic: Combined manual data with synthetically generated instances using a negation library and paraphrasing.
  • Train_Full-Synthetic: A larger synthetic dataset generated by negating entailment instances and paraphrasing original statements multiple times.

Key Insights and Ideas

  • Mistral-7B's Potential: The Mistral-7B model, even in a quantized form, proved to be a strong foundation for this specialized NLI task. Its ability to handle long contexts was a significant advantage.
  • Data Augmentation is Crucial: The performance gains observed with augmented datasets (especially "Train_Manual-Synthetic") highlight the importance of sufficient and diverse training data for LLMs in specialized domains. Simply relying on the original dataset was insufficient for optimal generalization.
  • Prompt Engineering Matters: The paper underscores that even with a capable LLM and augmented data, the prompt design is critical. A well-crafted prompt can significantly influence the model's ability to understand and execute the task correctly.
  • Faithfulness vs. Accuracy Trade-off: While the model achieved top-tier classification accuracy (macro F1-score), it struggled with faithfulness and consistency. This suggests that models might learn "shortcuts" to achieve high scores without truly understanding the underlying reasoning, a common challenge in LLM evaluation.

Key Results

The authors' best submission, trained on the "Train_Manual-Synthetic" dataset, achieved:

  • Macro F1-score: 0.80 (1st place on the leaderboard)
  • Consistency score: 0.72 (15th place)
  • Faithfulness score: 0.83 (11th place)

This indicates excellent performance in correctly classifying entailment/contradiction but highlights a gap in ensuring the model's reasoning is sound and robust to minor statement alterations.

Limitations and Future Work

The authors acknowledge several limitations and propose future directions:

  • Faithfulness and Consistency: The model's performance on these metrics was significantly lower than its classification accuracy, indicating a need for further research into improving robust reasoning.
  • Data Augmentation Quality: While synthetic data augmentation helped, the authors note that the quality of automatically generated examples needs careful control. The lack of explicit augmentation for "text appended" errors, for instance, proved costly.
  • Future Work:
    • Exploring other medical-domain-specific LLMs (e.g., qCammel, BioMistral).
  • Investigating advanced prompt optimization techniques beyond manual curation.
  • Incorporating more diverse data from related tasks or sources.
  • Focusing on curating training sets that specifically target statement interventions (paraphrasing, contradiction) rather than just increasing the quantity of base statements.

Summary updated

BibTeX
@inproceedings{guimaraes2024lisbon,
  author = {Artur Guimarães and Bruno Martins and João Magalhães},
  title = {{Lisbon Computational Linguists at SemEval-2024 Task 2: Using a Mistral-7B Model and Data Augmentation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure