MDC at SemEval-2023 Task 7: Fine-tuning Transformers for Textual Entailment Prediction and Evidence Retrieval in Clinical Trials

Robert Bevan, Oisín Turbitt, Mouhamad AboshokorACL 2023

TL;DR

This paper tackles the challenge of automatically extracting and verifying information from the vast and growing body of clinical trial reports. The authors developed a two-stage pipeline using transformer models to first retrieve relevant evidence and then predict whether a given hypothesis is supported by that evidence, achieving competitive results in the SemEval-2023 Task 7.

What Problem Does This Paper Address?

The sheer volume of clinical trial data available online presents a significant hurdle for researchers and medical practitioners. Manually sifting through hundreds of thousands of trial reports to find specific information or verify claims is time-consuming and impractical. This paper addresses the need for automated systems that can efficiently extract key information and assess the validity of hypotheses based on this data, thereby accelerating the systematic review process and the translation of experimental medicines.

Methods

The core of the proposed solution is a two-stage pipeline leveraging pre-trained transformer models.

  1. Evidence Retrieval: The first stage focuses on identifying sentences within clinical trial reports that are relevant to a given hypothesis. The authors fine-tuned PubMedBERT, a transformer model pre-trained on biomedical literature, for this task. The model was trained to predict the relevance of individual tokens, which were then aggregated to determine the relevance of entire lines.
  2. Textual Entailment: The second stage takes the retrieved evidence and the original hypothesis and predicts whether the hypothesis is true, false, or neutral (entailment). For this, they fine-tuned BioLinkBERT, another domain-specific transformer model known for its strong performance on biomedical NLP tasks. The output of the evidence retrieval model was fed as input to the textual entailment model.

A key methodological consideration was the token limit of transformer models (typically 512 tokens). To mitigate this, the authors prioritized evidence retrieval to filter the input text for the entailment model. They also experimented with different methods for converting token-level relevance predictions to line-level relevance for the evidence retrieval task.

Datasets

The primary dataset used is the NLI4CT dataset (Jullien et al., 2023). This dataset comprises:

  • 1000 clinical trial records.
  • 2400 hypotheses related to these trials.
  • Labels indicating relevant trial record sections, relevant subsection evidence lines, and entailment labels (true/false).

The dataset is divided into training, development, and test sets. The authors note that the training and development sets overlap in terms of clinical trials, while the test set includes additional trials.

Key Insights and Ideas

  • Pipeline Approach is Effective: The two-stage pipeline (evidence retrieval followed by textual entailment) proved to be a robust strategy, particularly for handling the token length limitations of transformer models. By first filtering for relevant evidence, they significantly reduced the amount of text that needed to be processed by the entailment model.
  • Evidence Retrieval is Crucial: The performance of the textual entailment model is heavily dependent on the quality of the evidence retrieved. The authors' evidence retrieval model achieved a strong F1 score, indicating its effectiveness in identifying relevant information.
  • Section-Specific Performance Variation: Evidence retrieval performance varied significantly across different sections of clinical trial reports (e.g., high performance for "Intervention" but lower for "Eligibility Criteria"). This highlights the inherent differences in how information is presented and the complexity of extracting evidence from various report sections.
  • Token-Level vs. Line-Level Prediction: While the evidence retrieval model was trained at the token level, converting these predictions to line-level relevance required careful consideration. The authors found that predicting a line as relevant if any of its constituent tokens were predicted as relevant worked best, though the difference between conversion strategies was minimal.

Key Results

  • Evidence Retrieval: The best performing evidence retrieval model achieved an F1 score of 0.804 on the test set, placing 10th in the competition. Performance varied by section, with an F1 score of 0.994 for "Intervention" and 0.586 for "Eligibility Criteria."
  • Textual Entailment: The textual entailment model, using the output of the evidence retrieval model, achieved an F1 score of 0.695 on the test set, placing 9th. The entailment model performed equally well for hypotheses concerning single clinical trials and pairs of trials.

Limitations and Future Work

  • Token Limit Still an Issue: Despite the pipeline approach, approximately 5% of examples still exceeded the token limit for the textual entailment model. These were truncated, which the authors acknowledge as a limitation.
  • Future Work - Strided Processing/Longer Attention: To address the token limit, they suggest exploring techniques like strided processing (averaging outputs from overlapping segments) or using models with longer attention spans.
  • Future Work - Single Model Approach: The authors propose investigating a single model trained to perform both evidence retrieval and textual entailment simultaneously. They believe this could be more efficient and potentially more robust by allowing the model to learn from both textual entailment signals and explicit evidence labels.
  • Future Work - Handling Mislabelled Evidence: A single model might also be better equipped to handle potential mislabelled evidence data in the training set.

Summary updated

BibTeX
@inproceedings{bevan2023mdc,
  author = {Robert Bevan and Oisín Turbitt and Mouhamad Aboshokor},
  title = {{MDC at SemEval-2023 Task 7: Fine-tuning Transformers for Textual Entailment Prediction and Evidence Retrieval in Clinical Trials}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure