YNU-HPCC at SemEval-2024 Task 2: Applying DeBERTa-v3-large to Safe Biomedical Natural Language Inference for Clinical Trials

Rengui Zhang, Jin Wang, Xuejie ZhangNAACL 2024

TL;DR

This paper tackles the challenge of automatically determining if a statement is entailed or contradicted by clinical trial reports, a crucial task for evidence-based medicine. The YNU-HPCC team adapted a powerful DeBERTa-v3-large language model, enhancing it with supervised contrastive learning and back-translation to improve its robustness against semantic interventions in the text. While achieving a respectable F1 score, the results highlight the ongoing need for deeper semantic comprehension in models.

What Problem Does This Paper Address?

The medical field generates a massive and ever-increasing volume of clinical trial reports (CTRs). Manually sifting through this literature to find relevant evidence for treatment decisions is becoming impractical. Natural Language Inference (NLI) offers a promising solution by automatically classifying the relationship between a given statement and one or more CTRs as either "entailment" (the statement is supported by the CTR) or "contradiction" (the statement is contradicted by the CTR).

The specific challenge in this task, SemEval-2024 Task 2, is "Safe Biomedical Natural Language Inference for Clinical Trials." This means not only must the model accurately infer the relationship, but it must also be robust to "semantic interventions" applied to the statements. These interventions are designed to test the model's true understanding, ensuring it doesn't rely on superficial cues. The goal is to ensure that the model's predictions are based on genuine semantic comprehension, not just pattern matching, which is vital for reliable clinical decision support.

Methods

The YNU-HPCC team's approach centers on a fine-tuned DeBERTa-v3-large model, a powerful pre-trained language model known for its strong performance on NLI tasks. They augmented this baseline with two key techniques:

  • Supervised Contrastive Learning (SCL): This technique aims to improve the model's ability to distinguish between similar and dissimilar text pairs. By pulling representations of similar samples closer together in the embedding space and pushing dissimilar ones apart, SCL encourages the model to learn more invariant features. This is particularly useful for handling the semantic interventions, as it should help the model focus on the core meaning rather than surface-level changes.
  • Back-translation: This data augmentation technique involves translating the training data into another language and then back to the original. This process effectively doubles the training data, introducing variations and potentially improving the model's robustness and generalization capabilities.

The overall loss function combines the standard cross-entropy loss (for accurate classification) with the supervised contrastive loss, balanced by hyperparameters $\alpha$ and $\beta$.

Datasets

The primary dataset used is NLI4CT, specifically designed for clinical NLI. This dataset consists of statements paired with Clinical Trial Reports (CTRs) focused on breast cancer research. The CTRs are structured into key sections: Eligibility Criteria, Intervention, Results, and Adverse Events. The NLI4CT dataset is divided into training, development, and test sets. The authors also mention using other NLI datasets for pre-training exploration, including MultiNLI, FeverNLI, ANLI, LingNLI, and WANLI.

Key Insights and Ideas

  • The importance of semantic interventions: The task's focus on interventions highlights a critical limitation of many NLI models – their tendency to exploit superficial patterns rather than truly understanding meaning. This paper's approach directly addresses this by aiming for invariant feature learning.
  • DeBERTa-v3-large as a strong foundation: The choice of DeBERTa-v3-large as the base model proved effective, demonstrating that larger models generally correlate with better performance.
  • SCL's potential for robustness: The authors hypothesized that SCL would help the model generalize better to data with semantic interventions. While the results showed improvement, the paper also suggests that parameter tuning and model depth might be crucial for fully realizing this potential.
  • Back-translation as a useful augmentation: Doubling the training data through back-translation was a straightforward yet effective way to increase the dataset size and potentially improve model performance.

Key Results

The YNU-HPCC team achieved a maximum F1 score of 0.77 on the development set, securing seventh position out of 32 teams. This performance was attributed to the combination of multi-task pre-training, supervised contrastive learning, and back-translation.

However, the paper also notes that the model exhibited suboptimal performance on Faithfulness (0.73) and Consistency (0.72) metrics on the test set. These metrics are crucial for evaluating how well the model's predictions align with the underlying semantics, especially after interventions. The lower scores on these metrics suggest that while the model could achieve a good F1 score, its reasoning might not always be sound, indicating a potential weakness in deep semantic comprehension.

Limitations and Future Work

The authors acknowledge several limitations:

  • Suboptimal performance on Faithfulness and Consistency: This indicates that the model's predictions might not always be based on valid reasoning, especially when dealing with interventions. The model may be too sensitive to minor input variations or fail to capture core semantic features accurately.
  • Potential underfitting: The authors suggest that suboptimal parameter settings or specific dataset characteristics might have led to underfitting, hindering the expected improvements from techniques like R-drop.
  • Insufficient model depth for causal reasoning: The paper points out that the model might lack the necessary depth for complex causal reasoning.
  • Truncated evidence from extended clinical trial premises: Long CTRs can pose challenges for models with fixed input lengths.

Future work should focus on:

  • Enhancing semantic comprehension and causal reasoning: Developing models with a deeper understanding of complex semantic relationships and causal links.
  • Refining contrastive learning: Further exploring and optimizing contrastive learning techniques to improve handling of complex data and interventions.
  • Addressing limitations of input length: Investigating methods to better process longer clinical trial premises.

Summary updated

BibTeX
@inproceedings{zhang2024ynu,
  author = {Rengui Zhang and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2024 Task 2: Applying DeBERTa-v3-large to Safe Biomedical Natural Language Inference for Clinical Trials}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
}
Figure