Stanford MLab at SemEval 2023 Task 7: Neural Methods for Clinical Trial Report NLI

Conner Takehana, Dylan Lim, Emirhan Kurtulus, Ramya Iyer, Ellie Tanimura, Pankhuri Aggarwal, Molly Cantillon, Alfred Yu, Sarosh Khan, Nathan ChiACL 2023

TL;DR

This paper presents a neural network system for natural language inference (NLI) on clinical trial reports (CTRs), specifically tackling entailment and evidence retrieval. Their approach, fine-tuning BERT-based models on clinical data and employing data augmentation, achieved a competitive ranking in the SemEval 2023 Task 7. This work highlights the challenges and potential of applying NLI to complex medical texts for better evidence-based care.

What Problem Does This Paper Address?

Clinical trial reports (CTRs) are crucial for understanding treatments and patient outcomes, but their sheer volume and complex, often variable, language make it difficult for clinicians to stay updated and provide personalized care. Natural Language Inference (NLI) models show promise in understanding relationships between texts, but applying them to medical data is challenging due to domain-specific jargon, varied scientific communication styles, and the need to distinguish subtle inferential relationships. This paper addresses the need for robust NLI systems that can accurately identify entailment/contradiction relationships and retrieve supporting evidence from CTRs, bridging the gap between research findings and clinical practice.

Methods

The core of the system involves fine-tuning pre-trained BERT-based language models on clinical data. Specifically, they experimented with several models, including Bio+Clinical BERT, RoBERTa Base, BioBERT, ELECTRA Small, and PubMedBERT. To address the limited size of the training dataset (1700 CTRs for training, 200 for evaluation), they employed several data augmentation techniques:

  • Back Translation: Translating text to a foreign language and back to create syntactically and semantically varied CTRs.
  • Synonym Replacement: Replacing words with their synonyms using NLTK and WordNet, leveraging BioWordVec and BioSentVec embeddings.
  • Random Perturbation: Introducing random insertions, deletions, and swaps of words to improve model robustness.

For Task 1 (entailment), they used a voting ensembling approach, combining the predictions of the top five performing models. For Task 2 (evidence retrieval), they implemented the BM25Okapi algorithm to extract supporting sentences from CTR premises. They also explored different optimizers (cross-entropy loss and Sharpness-Aware Minimization - SAM) and learning rates.

Datasets

The primary dataset used for training and evaluation is from SemEval Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. This dataset consists of 2400 statements collected by clinical trial organizers, domain experts, and research oncologists. Each data point includes a pair of CTRs, a statement, a section marker, and an entailment/contradiction label. The paper also mentions using the MIMIC-III clinical database for pre-training some of the BERT models.

Key Insights and Ideas

  • Domain-Specific Models Shine: BERT-based models pre-trained on medical or clinical data (like Bio+Clinical BERT and PubMedBERT) consistently outperformed general-domain models, underscoring the importance of domain-specific knowledge for NLI in healthcare.
  • Data Augmentation is Crucial: With a small training set, data augmentation techniques like back translation and synonym replacement were vital for improving model generalization and performance.
  • Ensembling Boosts Performance: Combining the predictions of multiple well-performing models through voting ensembling led to a higher F1 score than any individual model, demonstrating the benefit of diverse model perspectives.
  • Learning Rate Sensitivity: The study highlighted that models can be highly sensitive to learning rates, with optimal rates varying significantly and some models failing to learn at all with certain rates.
  • Evidence Retrieval is Challenging but Promising: While the BM25 baseline showed effectiveness in retrieving relevant evidence, there's clear room for improvement in its F1 score, suggesting that more sophisticated evidence retrieval methods could further enhance the system's utility.

Key Results

  • The top-performing individual models for Task 1 included PubMedBERT (F1 = 0.660), RoBERTa Base (F1 = 0.658), and BioBERT Disease (F1 = 0.655).
  • The voting ensembled model achieved an F1 score of 0.662 for Task 1, outperforming individual models and placing the system 20th out of 30 submissions.
  • For Task 2 (evidence retrieval), the BM25 baseline achieved a precision of 0.522748, recall of 0.622034, and an F1 score of 0.461280, indicating it could retrieve a majority of relevant evidence but with room for improvement.
  • Cross-entropy loss optimizer outperformed the SAM optimizer by 11.5 accuracy percentage points in some experiments.
  • A learning rate of 10⁻⁴ was found to be most successful for many of the top-performing models.

Limitations and Future Work

  • Data Augmentation Concerns: The authors acknowledge that the synthesized augmented data might introduce noise or skewed results, as no strict regulation was performed on its quality.
  • Limited Hyperparameter Testing: For Task 1, they were only able to perform one run with a 10⁻² learning rate and only tested the SAM optimizer with one specific model (Bio+Clinical BERT). Further testing of these hyperparameters is recommended.
  • Future Work Suggestions:
    • Generating word embeddings with BioWordVec and comparing them to GloVe embeddings.
  • Exploring more advanced data augmentation techniques.
  • Investigating other optimizers and hyperparameter tuning strategies.
  • Developing more sophisticated evidence retrieval methods beyond BM25.

Summary updated

BibTeX
@inproceedings{takehana2023stanford,
  author = {Conner Takehana and Dylan Lim and Emirhan Kurtulus and Ramya Iyer and Ellie Tanimura and Pankhuri Aggarwal and Molly Cantillon and Alfred Yu and Sarosh Khan and Nathan Chi},
  title = {{Stanford MLab at SemEval 2023 Task 7: Neural Methods for Clinical Trial Report NLI}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure