I2R at SemEval-2023 Task 7: Explanations-driven Ensemble Approach for Natural Language Inference over Clinical Trial Data

Saravanan Rajamanickam, Kanagasabai RajaramanACL 2023

TL;DR

This paper presents an "explanation-driven" ensemble approach for Natural Language Inference (NLI) specifically tailored for Clinical Trial Reports (CTRs). By first identifying relevant evidence and then performing inference on that evidence, their T5-based system achieved strong results in the SemEval-2023 Task 7, ranking 8th for both textual entailment and evidence retrieval.

What Problem Does This Paper Address?

Natural Language Understanding (NLU) research has long focused on Natural Language Inference (NLI), the task of determining if a hypothesis is entailed, contradicted, or neutral given a premise. While NLI is fundamental, applying it to complex, lengthy, and domain-specific texts like Clinical Trial Reports (CTRs) presents unique challenges. CTRs are often much longer than typical NLI datasets, potentially exceeding the token limits of even large language models. Furthermore, the specialized terminology and unstructured nature of clinical data make accurate inference difficult. This paper tackles the SemEval-2023 Task 7, which specifically aims to perform NLI over CTRs, requiring not only to determine the inference relation but also to extract supporting evidence.

Methods

The authors propose an "explanation-driven" NLI approach. The core idea is to break down the complex NLI task into two stages:

  1. Evidence Classification: Given a CTR premise and a statement, they first identify sentences within the CTR that are "relevant" to the statement. This is framed as a binary classification problem where a T5 model is trained to distinguish between relevant and irrelevant evidence.
  2. Evidence-level NLI: For the identified relevant evidence, they perform a 3-way NLI (entailment, contradiction, neutral). This is also handled by a T5 model, treating the statement as the hypothesis and the relevant evidence as the premise. Irrelevant sentences are implicitly labeled as "neutral."

Finally, an ensemble approach is used to combine the evidence-level inferences to arrive at the final prediction for both tasks. For Task 1 (Textual Entailment), they ensemble the NLI labels based on the ratio of contradiction to entailment labels. For Task 2 (Evidence Retrieval), they leverage the multi-task learning capabilities of T5 to directly retrieve supporting facts. They experimented with various BERT and T5 models, ultimately finding T5 base to be the most effective.

Datasets

The primary dataset used is from the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data. This dataset consists of 2,400 statements, evenly split across different sections and classes, derived from breast cancer CTRs. Each CTR is broken down into sections like Eligibility Criteria, Intervention, Results, and Adverse Events. The dataset is designed for two sub-tasks: Textual Entailment and Evidence Retrieval.

Key Insights and Ideas

  • Explanation-driven NLI is effective for complex domains: By focusing on identifying and reasoning over specific pieces of evidence, the approach mitigates the challenges posed by long and complex CTR documents.
  • T5 excels in this domain: The T5 model, particularly the T5 base, demonstrated superior performance compared to BERT-based models, suggesting its suitability for this type of structured text-to-text reasoning.
  • Evidence retrieval is crucial for interpretability: The ability to extract supporting facts (Task 2) not only aids in justifying the NLI prediction but also makes the system's reasoning more transparent, which is vital in clinical contexts.
  • Handling "empty" evidence is a challenge: The authors identified that situations where no relevant evidence is found (due to lengthy sections or domain-specific jargon) can lead to errors, necessitating rule-based adjustments for exceptions.

Key Results

The authors' final T5 base model achieved the following results on the SemEval-2023 Task 7:

  • Task 1 (Textual Entailment): F1 score of 70.1% (ranked 8th).
  • Task 2 (Evidence Retrieval): F1 score of 80.2% (ranked 8th).

They were one of only five systems to rank within the Top 10 for both tasks, highlighting the robustness of their approach.

Limitations and Future Work

The authors acknowledge that their system achieved low precision but high recall for Task 1, indicating a tendency to over-predict entailment. They also note that the model struggled in situations where no relevant evidence could be found, leading them to develop manual rules to handle exceptions and negations.

For future work, they suggest:

  • Exploring the generative text-to-text framework of T5 for NLI, potentially combined with arithmetic reasoning.
  • Investigating the impact of pre-training data on domain-specific clinical corpora to further enhance performance. They specifically mention using models like SciFive and BioBERT as promising directions.

Summary updated

BibTeX
@inproceedings{rajamanickam2023i,
  author = {Saravanan Rajamanickam and Kanagasabai Rajaraman},
  title = {{I2R at SemEval-2023 Task 7: Explanations-driven Ensemble Approach for Natural Language Inference over Clinical Trial Data}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure