TüDuo at SemEval-2024 Task 2: Flan-T5 and Data Augmentation for Biomedical NLI

Veronika Smilga, Hazem AlabiadNAACL 2024

TL;DR

This paper explores data augmentation for biomedical Natural Language Inference (NLI) using smaller language models (Flan-T5) and GPT-3.5-Turbo. While augmentation improved model robustness (faithfulness and consistency), it slightly decreased performance on the original dataset distribution. The best system, a fine-tuned Flan-T5 XL, achieved a top-10 ranking in the SemEval-2024 Task 2.

What Problem Does This Paper Address?

The paper tackles the challenge of building robust Natural Language Inference (NLI) systems for the complex and heterogeneous data found in Clinical Trial Reports (CTRs). Specifically, it addresses the need for models that are not only accurate but also "faithful" (correctly change predictions for semantically different inputs) and "consistent" (give the same predictions for semantically preserving changes). This is crucial in biomedical fields where errors can have high costs. The authors also highlight the potential biases in large language models and the cost-inefficiency of using proprietary models, advocating for the use of smaller, fine-tuned models on curated datasets.

Methods

The core of the paper involves fine-tuning Flan-T5 models (ranging from Small to XL, under 3 billion parameters) on the NLI4CT-2024 dataset. They investigate the impact of data augmentation generated by GPT-3.5-Turbo. The augmentation techniques include:

  • Synonym-based semantic-preserving changes: Replacing words with synonyms.
  • Syntactic semantic-preserving changes: Altering sentence structure without changing meaning.
  • Random fact addition semantic-preserving changes: Appending a true random fact.
  • Semantic-altering changes: Formulating a hypothesis that contradicts the original.

They evaluate models in zero-shot, few-shot, and fine-tuned settings. For fine-tuning, they use a combination of original and augmented data, employing techniques like LoRA for larger models due to computational constraints.

Datasets

  • NLI4CT-2024 Dataset: This dataset consists of premise-hypothesis pairs derived from breast cancer Clinical Trial Reports (CTRs). The premises are drawn from different sections of CTRs (Intervention, Eligibility, Results, Adverse Events), introducing complexity due to varying lengths, numerical data presence, and required world knowledge.
  • Augmented Data: The authors generated a significant amount of augmented data using GPT-3.5-Turbo, expanding the original training set by 4.5 times. This augmented data was used to train their models.
  • EQUATE Dataset: Used for pre-fine-tuning a specialized model for the "Adverse Events" section to enhance numerical inference capabilities.

Key Insights and Ideas

  • Smaller models can be effective: Despite the rise of massive LLMs, smaller models like Flan-T5 can achieve competitive performance on specific tasks, especially when fine-tuned.
  • Data augmentation is a double-edged sword: While augmentation significantly improved model robustness (faithfulness and consistency), it led to a slight decrease in performance (F1 score) on the original dataset distribution. This suggests a trade-off between generalization and performance on the exact training distribution.
  • Faithfulness and Consistency are key for robustness: The new metrics introduced in SemEval-2024 Task 2 (faithfulness and consistency) are crucial for evaluating model robustness, especially in sensitive domains like healthcare.
  • Semantic-altering changes boost faithfulness: Specifically, generating contradictory hypotheses was the most effective way to improve model faithfulness.
  • Semantic-preserving changes boost consistency: Synonym replacement and random fact addition were particularly effective in improving model consistency.

Key Results

  • The best-performing system, a Flan-T5 XL model fine-tuned on a combination of original and over 6,000 augmented examples, ranked in the top 10 for all three metrics (F1, faithfulness, and consistency) in SemEval-2024 Task 2.
  • Individual augmentation analysis showed that synonym-based augmentation had a positive effect on F1, while other types led to a decrease.
  • All augmentation types increased consistency. Random fact addition yielded the highest increase in consistency.
  • Semantic-altering changes led to a more faithful model, while semantic-preserving changes decreased faithfulness.
  • The final model achieved a faithfulness of 0.841 and consistency of 0.752, with an F1 score of 0.76, compared to the original data-only model's faithfulness of 0.780, consistency of 0.667, and F1 of 0.779.

Limitations and Future Work

  • Performance drop on original distribution: A key limitation is that data augmentation, while improving robustness, slightly decreased the F1 score on the original dataset.
  • Section-specific performance: The paper notes that performance varies across different sections of CTRs, with the "Adverse Events" section (often containing more numbers) showing lower F1 scores.
  • Specialized model for Adverse Events: An attempt to create a specialized model for the "Adverse Events" section using pre-fine-tuning on EQUATE did not yield better results on the test set, suggesting that training on a smaller subset of data might hinder robustness.
  • Future work could explore: Further investigation into the optimal balance between augmentation and original data, and more sophisticated methods for generating semantically altering data.

Summary updated

BibTeX
@inproceedings{smilga2024tuduo,
  author = {Veronika Smilga and Hazem Alabiad},
  title = {{TüDuo at SemEval-2024 Task 2: Flan-T5 and Data Augmentation for Biomedical NLI}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure