BEEDS: Large-Scale Biomedical Event Extraction using Distant Supervision and Question Answering

Xing David Wang, Ulf Leser, Leon WeberACL 2022

TL;DR

This paper introduces BEEDS, a novel system for automatically extracting complex biomedical event structures from vast amounts of literature. By framing event extraction as a question-answering task and leveraging distant supervision, BEEDS significantly improves recall in identifying crucial biological interactions, paving the way for more comprehensive pathway knowledge bases.

What Problem Does This Paper Address?

The sheer volume of biomedical literature makes it impossible for human experts to manually curate all the important biological events and interactions that govern cellular processes. Existing pathway knowledge bases, while invaluable, are notoriously incomplete due to the manual effort required for their construction. This incompleteness hinders research in areas like network analysis and enrichment analysis, which rely on accurate and exhaustive lists of biochemical reactions. The challenge lies in automatically extracting these complex event structures (like enzyme-substrate-site interactions) from unstructured text at scale, especially when labeled training data is scarce.

Methods

BEEDS tackles this problem with a three-step pipeline:

  1. Document Retriever: This component first identifies documents likely to contain relevant information for a given query. It uses the BM25 algorithm, a standard but effective method for text retrieval, to find documents from a massive corpus (PubMed abstracts and full texts). To improve retrieval, it expands protein queries with synonyms from NCBI Gene and includes additional event triggers.
  2. Document Reader: This is the core of the event extraction. BEEDS employs a transformer-based model (specifically, SciBERT, a BERT variant pre-trained on scientific text) to identify and extract answer spans within the retrieved documents. This reader is trained using a question-answering paradigm, where questions are generated from known event structures.
  3. Entity Normalizer: After extracting potential event participants, this component maps them to canonical database identifiers (like NCBI Entrez Gene IDs or UniProt IDs). This step is crucial for integrating extracted information into structured knowledge bases.

A key innovation is BEEDS' use of distant supervision. Instead of relying solely on scarce, manually annotated data, BEEDS generates noisy training examples by transforming existing knowledge base events into question-answer pairs. This allows the model to learn from a much larger, albeit imperfect, dataset. They also incorporate a multi-instance learning approach, assuming that within a "bag" of noisy annotations, at least one is correct, making the training more robust.

Datasets

BEEDS utilizes two main types of datasets:

  • Knowledge Base (KB) Data Set: This is a distantly supervised dataset derived from the union of seven pathway databases (KEGG, PID, Reactome, HumanCyc, INOH, PANTHER, and NetPath). Events from these databases are converted into question-answer pairs.
  • BioNLP Data Set: This is a directly supervised dataset containing gold-standard annotations from the GENIA and Pathway Curation challenges.

Key Insights and Ideas

  • Question Answering as a Powerful Framework: Framing event extraction as a question-answering task, especially with multi-turn questions, allows BEEDS to uncover more complex, multi-participant events that simpler relation extraction systems might miss.
  • Distant Supervision is Crucial for Scale: The scarcity of manually labeled biomedical data is a major bottleneck. BEEDS demonstrates that leveraging existing knowledge bases through distant supervision can significantly boost performance and enable large-scale extraction.
  • Multi-Instance Learning Mitigates Noise: The noisy nature of distantly supervised data is a challenge. Multi-instance learning helps the model learn effectively by assuming at least one annotation in a group is correct, rather than requiring all to be perfect.
  • Entity Normalization is Key for Integration: Simply extracting text spans isn't enough. Mapping these entities to canonical identifiers is essential for building usable knowledge bases and comparing findings across different sources.

Key Results

  • Improved Recall: BEEDS achieves a higher overall recall than the strong EVEX baseline, particularly for post-translational modification (PTM) events involving enzyme-substrate-site triples (a ~5 percentage point increase).
  • Multi-Turn Event Extraction: BEEDS significantly outperforms EVEX in extracting multi-turn questions (complex events), achieving a recall of 14.30% compared to EVEX's near-zero recall.
  • Efficiency: Despite higher recall, BEEDS makes fewer predictions than EVEX, suggesting a more focused and accurate extraction process.
  • Sample Precision: While BEEDS' overall recall is higher, its sample precision (49.09%) is lower than EVEX's (63.30%), likely due to the inherent noise in distant supervision and error propagation in multi-turn events.
  • Impact of Retrieval Size: Increasing the retrieval size (r) from 100 to 1,000 documents during evaluation almost doubles the knowledge base recall, highlighting the importance of thorough document retrieval.
  • Benefit of Directly Supervised Data: Adding directly supervised data to the training set significantly boosts performance, increasing recall from 4.84% to 65.23% on the BioNLP dataset.

Limitations and Future Work

  • Sample Precision: BEEDS' sample precision is lower than EVEX's, suggesting room for improvement in reducing false positives, especially in complex, multi-turn events. The authors hypothesize this is due to error propagation.
  • Normalizer Optimization: The current normalizer shows potential for improvement, as a simpler dictionary lookup could increase recall by about a third, indicating that the existing normalizer might be missing some valid mappings.
  • Future Directions: The authors plan to explore other retrieval approaches, such as dense retrieval methods, to potentially improve performance further. They also suggest investigating how other state-of-the-art biomedical event extraction methods, like DeepEventMine, scale to large-scale curation efforts and comparing them to BEEDS. Further work on handling event modifiers (like negation or speculation) and event polarities is also implied.

Summary updated

BibTeX
@inproceedings{wang2022beeds,
  author = {Xing David Wang and Ulf Leser and Leon Weber},
  title = {{BEEDS: Large-Scale Biomedical Event Extraction using Distant Supervision and Question Answering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure