lasigeBioTM at SemEval-2023 Task 7: Improving Natural Language Inference Baseline Systems with Domain Ontologies

Sofia I. R. Conceição, Diana F. Sousa, Pedro Silvestre, Francisco M CoutoACL 2023

TL;DR

This paper tackles the challenge of Natural Language Inference (NLI) in the complex domain of Clinical Trial Reports (CTRs). The authors demonstrate that augmenting standard NLI systems with domain-specific ontologies significantly improves their ability to infer relationships between hypotheses and CTR text, a crucial step for extracting valuable health information.

What Problem Does This Paper Address?

Extracting reliable information from Clinical Trial Reports (CTRs) is vital for advancing medical knowledge. However, CTRs are dense with specialized terminology, acronyms, and abbreviations that can lead to ambiguity, making it difficult for Natural Language Inference (NLI) systems to accurately determine if a given hypothesis can be inferred from the text. For instance, an acronym like "ALS" could refer to "Amyotrophic Lateral Sclerosis" or a gene symbol, requiring contextual knowledge for correct interpretation. This paper aims to improve NLI performance in this challenging domain by incorporating external domain knowledge.

Methods

The team, lasigeBioTM, participated in SemEval-2023 Task 7, Subtask 1: Textual Entailment. Their approach combines several techniques:

  • Ontology Integration: They leverage domain ontologies (e.g., BioAssay Ontology, Human Disease Ontology) to enrich the text. This involves identifying entities in the CTRs and linking them to concepts within these ontologies.
  • NLP Techniques: Standard NLP tools like tokenization and named-entity recognition (NER) are used, specifically the scispaCy library with a biomedical model.
  • Shortest Dependency Path (SDP): To reduce noise and focus on relevant relationships, the SDP between annotated entities is extracted.
  • Rule-Based Approaches: Rules are developed to handle numerical comparisons and measurements within the text, which are crucial for clinical data.
  • Ancestors from Ontologies: By traversing the ontology hierarchy, ancestors of identified entities are retrieved to provide broader contextual information.

The core idea is to combine these enriched features (annotations, SDP, ancestors, rules) with the original text section to improve the similarity calculation between the statement and the CTR.

Datasets

The primary dataset used is from the SemEval-2023 Task 7: Multi-evidence Natural Language Inference for Clinical Trial Data (NLI4CT) dataset. This dataset consists of statements and corresponding Clinical Trial Reports (CTRs), with the task of determining an entailment or contradiction relationship. The CTRs are divided into sections like eligibility criteria, intervention, results, and adverse events.

Key Insights and Ideas

  • Ontologies are Powerful for NLI: The study strongly suggests that domain ontologies are not just helpful but essential for improving NLI performance in specialized domains like clinical trials. They provide the necessary background knowledge to resolve ambiguities.
  • Combining Multiple Ontologies is Beneficial: The paper highlights the use of eight different ontologies, indicating that a diverse set of domain knowledge sources can capture a wider range of medical concepts.
  • Shortest Dependency Path Can Be Sensitive: While SDP helps reduce noise, the authors found that for very short statements, the SDP might not capture enough relevant information, leading to worse results than using just the statement text.
  • Rule-Based Systems Still Relevant: Despite the rise of deep learning, rule-based approaches for handling specific data types like numerical measurements remain valuable and contribute to overall performance.

Key Results

The lasigeBioTM system achieved an overall ranking of 21st in the competition, with an F1-score of 0.661, precision of 0.511, and recall of 0.936. The authors note that the high recall is partly due to the binary nature of the task and the model's tendency to predict entailment. Ablation studies showed that each component of their system (No Annotated CTRs, No SDP, No Count Rules) individually improved upon the baseline NLI4CT Starter Kit, with the full pipeline achieving the highest scores.

Limitations and Future Work

The authors acknowledge that their approach to handling comparison statements (joining text from multiple CTRs) led to noise and a loss of context. They also noted that the SDP approach can sometimes fail if the statement is too short. For future work, they propose exploring thresholds for non-specific annotations and ancestors to generate more meaningful annotations and potentially improve the handling of comparison statements.

Summary updated

BibTeX
@inproceedings{conceicao2023lasigebiotm,
  author = {Sofia I. R. Conceição and Diana F. Sousa and Pedro Silvestre and Francisco M Couto},
  title = {{lasigeBioTM at SemEval-2023 Task 7: Improving Natural Language Inference Baseline Systems with Domain Ontologies}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}
Figure