Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports

Juraj Vladika, Florian MatthesACL 2023

TL;DR

This paper tackles the challenge of automatically extracting actionable insights from the ever-growing volume of clinical trial reports. The authors developed a novel ensemble system that combines evidence retrieval and natural language inference, achieving a 3rd place ranking in the SemEval-2023 Task 7 competition. This work offers a significant step towards automating the synthesis of medical knowledge, aiding researchers and clinicians.

What Problem Does This Paper Address?

The sheer volume of clinical trial reports (CTRs) published daily makes it nearly impossible for medical professionals to keep up with the latest findings. This deluge of information hinders the timely integration of new evidence into healthcare practices. To address this, researchers are turning to Natural Language Processing (NLP) to automate the process of extracting relevant information and inferring logical conclusions from these complex documents. SemEval-2023 Task 7 specifically focused on two intertwined subtasks: 1) Evidence Retrieval, identifying sentences within a CTR that support a given claim, and 2) Natural Language Inference (NLI), determining if a claim is entailed or contradicted by the information in the CTR, using the retrieved evidence.

Methods

The authors proposed two main system architectures:

  • Pipeline System: This approach treats the two subtasks sequentially. First, an evidence retrieval model identifies relevant sentences from the clinical trial report. Then, a separate NLI model uses these selected sentences to determine the logical relationship (entailment or contradiction) between the claim and the evidence.
  • Joint System: This more integrated approach uses multi-task learning to train a single model that simultaneously learns to perform both evidence retrieval and NLI. The idea is that these tasks are mutually beneficial: good evidence selection aids inference, and the inference task can guide the selection of more pertinent evidence.

The final system is an Ensemble System that combines the outputs of the best pipeline and joint systems. This is achieved by averaging their predicted probabilities, weighted to optimize performance. This ensemble approach leverages the strengths of both individual systems to achieve a more robust and accurate outcome.

Datasets

The primary dataset used for this task is derived from ClinicalTrials.gov, a public database of clinical studies. The dataset specifically focuses on breast cancer trials and includes:

  • 1,200 clinical trial reports.
  • 2,400 claims related to these reports, annotated by domain experts.

The claims are categorized into four types: Eligibility Criteria, Intervention, Results, and Adverse Events. The dataset was split into training (1,700 claims), development (200 claims), and a hidden test set (500 claims).

Key Insights and Ideas

  • Joint learning is powerful: The joint system, which learns evidence retrieval and NLI simultaneously, generally outperformed the pipeline system, highlighting the synergistic relationship between these tasks.
  • Domain-specific models excel: Fine-tuned BERT models (like ClinicalBERT and BioBERT) significantly outperformed the general BERT model, demonstrating the value of pre-training on specialized biomedical text.
  • Structured knowledge matters: UmlsBERT, which incorporates structured knowledge from the UMLS metathesaurus, showed strong performance, suggesting that integrating external medical knowledge is beneficial.
  • DeBERTa-v3 is a top performer: The DeBERTa-v3 model emerged as the best-performing base model, particularly excelling in the textual entailment task, underscoring its advanced architecture for NLU tasks.
  • Ensembling boosts performance: Combining the pipeline and joint systems through an ensemble approach yielded the best overall results, a common strategy for achieving state-of-the-art performance in NLP competitions.
  • Post-processing is crucial: Truncating selected evidence sentences to a maximum of 20 was found to be important for improving precision, especially for claims involving quantifiers.

Key Results

The authors' ensemble system achieved a 3rd place ranking out of 40 participating teams in the SemEval-2023 Task 7 competition.

  • On the development set, the best pipeline system achieved an F1 score of 79.8 for evidence retrieval and 77.2 for textual entailment.
  • The best joint system achieved F1 scores of 80.4 for evidence retrieval and 78.3 for textual entailment.
  • The final ensemble system achieved the highest scores, with an F1 of 81.8 for evidence retrieval and 79.8 for textual entailment on the test set.

Limitations and Future Work

The authors acknowledge several challenges and suggest future directions:

  • Commonsense Reasoning: The model struggles with claims requiring everyday knowledge and innate human judgments.
  • Numerical Reasoning: Applying basic mathematical operations and understanding comparisons remains a challenge.
  • Multi-hop Reasoning: Combining multiple pieces of evidence to reach a conclusion is difficult for the current system.
  • Medical and World Knowledge: The system could be improved by incorporating more extensive medical and general world knowledge.

Future work could focus on addressing these specific reasoning challenges and further enhancing the system's ability to leverage external knowledge sources.

Summary updated

BibTeX
@inproceedings{vladika2023sebis,
  author = {Juraj Vladika and Florian Matthes},
  title = {{Sebis at SemEval-2023 Task 7: A Joint System for Natural Language Inference and Evidence Retrieval from Clinical Trial Reports}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure