Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles

Oumaima El Khettari, Noriki Nishida, Shanshan Liu, Rumana Ferdous Munne, Yuki Yamagata, Solen Quiniou, Samuel Chaffron, Yuji MatsumotoACL 2024

TL;DR

This paper introduces a novel two-stage system for extracting structured biomedical knowledge from text, specifically focusing on identifying and relating biological processes without relying on explicit mentions of these processes in the text. They also release the HOIP dataset, a new benchmark for this "mention-agnostic" information extraction task.

What Problem Does This Paper Address?

Biomedical research generates a massive amount of data, and efficiently extracting knowledge from this data is crucial for advancing science and healthcare. Traditional information extraction (IE) methods often rely on identifying explicit mentions of entities (like biological processes) in text. However, in many real-world scenarios, especially in complex biomedical literature, these processes might be implicitly described or inferred rather than explicitly stated. This lack of explicit mentions poses a significant challenge for existing IE systems. The authors aim to overcome this by developing a "mention-agnostic" approach that can identify and relate biological processes even when they are not directly named in the text.

Methods

The proposed system is a two-stage pipeline:

  1. Process Identification: This stage identifies biological process entities within a given passage. Crucially, it does this without requiring explicit mentions. The authors explore two approaches:
  2. BERT-based Supervised Approach: This frames the problem as a binary classification task for each potential process name against a passage.
  3. LLM-based In-Context Learning (ICL): This leverages large language models (LLMs) to generate process names based on prompts, exploring zero-shot and few-shot settings to assess performance in low-resource scenarios.
  4. Document-level Relation Extraction (DocRE): Once processes are identified, this stage determines the relationships between them. The authors adapt existing DocRE methods to be mention-agnostic. They propose two main approaches:
  5. QA-Style DocRE Model: This reformulates the task as a question-answering problem, generating questions for potential entity pairs and using a BERT encoder.
  6. Mention-Agnostic ATLOP (MA-ATLOP): This extends a popular DocRE method (ATLOP) to work without explicit mention spans. It encodes entities and passages jointly and then classifies relations between entity pairs. They also introduce Negative Entity Sampling (NES) to improve robustness against noisy entities.

Datasets

  • HOIP Dataset: A novel, manually annotated dataset created for this research. It's based on the Homeostasis Imbalance Process Ontology (HOIP) and contains passages from PubMed articles describing COVID-19 infectious processes. The dataset is designed to facilitate the development and benchmarking of mention-agnostic IE models.
  • CDR Dataset: Used for evaluating the DocRE component. This dataset consists of PubMed abstracts annotated with Chemical or Disease entities and their relations.

Key Insights and Ideas

  • Implicit Information is Key: The core idea is that biomedical knowledge is often implicit, and systems need to move beyond simple mention detection to truly understand complex biological processes.
  • LLMs for Low-Resource Settings: Generative LLMs show promise for process identification in low-resource, in-domain corpora like the HOIP dataset, especially when guided by effective few-shot demonstrations.
  • BERT's Strength in Relation Classification: While LLMs are useful for generation, fine-tuned BERT models demonstrate superior performance in relation classification, even without explicit mentions.
  • Mention-Agnostic DocRE is Challenging but Feasible: The MA-ATLOP model shows that it's possible to achieve competitive DocRE performance without relying on mention spans, though mentions still provide a significant advantage.
  • Error Propagation is a Concern: The cascaded nature of the two-stage system means errors in process identification can negatively impact relation extraction, highlighting the need for robust individual components.

Key Results

  • Process Identification: Generative LLMs (Llama3) showed strong performance, especially in few-shot settings, outperforming BERT-based models in some configurations.
  • Relation Classification (DocRE):
    • On the HOIP dataset, BERT-based models achieved significantly higher F1 scores (56.5-59.0) compared to LLMs (around 17.6) when using ground-truth entities.
  • On the CDR dataset, the mention-agnostic MA-ATLOP model achieved an F1 score of 62.3, which was lower than the mention-based ATLOP (69.7) but still competitive, especially considering the absence of mention spans.
  • The MA-ATLOP model consistently outperformed the QA-Model in DocRE tasks.
  • End-to-End System: The authors note that improvements in both process identification and relation classification are crucial for the overall system's performance, indicating room for future work.

Limitations and Future Work

  • Cascaded System Vulnerability: The two-stage pipeline is susceptible to error propagation, where errors in the first stage (process identification) negatively impact the second stage (relation extraction). The models are also disconnected and cannot interact.
  • Domain Specificity: The current methodology has only been evaluated on the HOIP ontology. Its performance on other biomedical domains and ontologies remains unknown.
  • Expert Evaluation: While expert analysis was performed, it was based on only two examples. More thorough evaluation by domain specialists is needed.
  • Future Work: The authors suggest exploring ways to improve recall and precision in both process identification and DocRE, potentially by developing techniques to avoid biases introduced by methods like Negative Entity Sampling and by enabling interaction between the two stages of the pipeline.

Summary updated

BibTeX
@inproceedings{khettari2024mention,
  author = {Oumaima El Khettari and Noriki Nishida and Shanshan Liu and Rumana Ferdous Munne and Yuki Yamagata and Solen Quiniou and Samuel Chaffron and Yuji Matsumoto},
  title = {{Mention-Agnostic Information Extraction for Ontological Annotation of Biomedical Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure