TL;DR
This paper presents a system for the SemEval-2023 Task 7, which focuses on natural language inference (NLI) for clinical trial data. The team developed a BioBERT-based model enhanced with supervised contrastive learning and back translation to accurately determine the entailment relationship between clinical trial reports and associated statements, achieving competitive results.
What Problem Does This Paper Address?
The ever-increasing volume of clinical trial reports (CTRs) makes it challenging for medical practitioners to stay updated and provide personalized, evidence-based care. Natural Language Inference (NLI) offers a promising avenue for large-scale interpretation and retrieval of medical evidence. Specifically, this work addresses the SemEval-2023 Task 7, Subtask 1, which requires determining if a given statement is entailed by or contradicts a clinical trial report. This task is crucial for extracting actionable insights from complex medical literature.
Methods
The core of the YNU-HPCC team's system is a BioBERT model, a BERT variant pre-trained on biomedical literature, which is more suitable for clinical text analysis than general-domain BERT. To improve classification performance, they incorporated two key techniques:
- Supervised Contrastive Learning: This method aims to pull samples with the same label closer together in the semantic space while pushing samples with different labels further apart. This enhances the model's ability to distinguish between entailment and contradiction.
- Back Translation: This data augmentation technique generates synthetic training data by translating sentences to another language and then back to the original. This process enriches the training dataset without altering the semantic meaning, thereby improving model robustness.
The system also explored other models like Bio_ClinicalBERT and Longformer, but BioBERT with the proposed enhancements yielded the best results.
Datasets
The primary dataset used is the NLI4CT shared task data set. This dataset consists of normalized JSON data derived from clinical trial reports (CTRs) and associated statements, explanations, and labels annotated by domain experts. The dataset includes training, development, and test sets, with specific sizes mentioned for each:
- Training set: 1700 expert-annotated CTRs.
- Development set: 200 samples.
- Test set: 500 samples.
- Clinical experiment report set: 999 samples.
The data is structured with information such as eligibility criteria, intervention details, results, and adverse events from the CTRs.
Key Insights and Ideas
- Domain-Specific Models Matter: Using BioBERT, pre-trained on biomedical texts, significantly outperforms general-domain BERT for clinical NLI tasks. This highlights the importance of domain adaptation in NLP.
- Contrastive Learning Boosts Discrimination: Supervised contrastive learning proved effective in enhancing the model's ability to differentiate between entailment and contradiction by explicitly learning to separate dissimilar samples.
- Data Augmentation is Key: Back translation effectively expanded the training data, leading to improved performance, especially when dealing with limited initial data.
- Text Length is a Major Hurdle: The inherent limitation of transformer models (like BioBERT) to process long sequences (512 tokens) poses a significant challenge for clinical trial reports, which can be very lengthy. Truncation leads to information loss, and while Longformer was explored, parameter tuning issues prevented it from achieving optimal results.
Key Results
The YNU-HPCC system achieved a Macro F₁-score of 0.679 on the test set for Subtask 1, ranking 12th place in the competition. On the development set, the BioBERT model with supervised contrastive learning and back translation achieved the best performance with an F₁-score of 0.665. The paper also shows that BioBERT generally outperforms BERT, and the combination of supervised contrastive learning and back translation yields substantial improvements over baseline models.
Limitations and Future Work
The authors acknowledge that the maximum input length limitation of transformer models is a significant constraint. Truncating long texts results in the loss of potentially crucial semantic information. While they experimented with Longformer to address this, they did not achieve the expected results due to GPU limitations and time constraints, suggesting that further exploration of appropriate parameter tuning for long-document models is needed. Future work could also involve exploring other biomedical text-processing models or more advanced text-length processing methods to further improve performance.
Summary updated