Tunable Domain-Independent Event Extraction in the MIRA Framework

Georgi Georgiev, Kuzman Ganchev, Vassil Momtchev, Deyan Peychev, Preslav Nakov, Angus RobertsNAACL 2009 · OpenReview

TL;DR

This paper presents a system for extracting molecular events from text, designed to be adaptable to different precision-recall trade-offs. By leveraging the MIRA framework and carefully tuning loss functions, they achieve tunable performance for event extraction, a crucial task for building biological knowledge bases.

What Problem Does This Paper Address?

Understanding molecular interactions is fundamental to biology. Large-scale efforts like the Pathway and Interaction Knowledge Base (PIKB) aim to consolidate this information, but automatically extracting these interactions from scientific literature remains a significant challenge. Specifically, the authors tackle the problem of event extraction, which involves identifying not just which molecules interact, but also the type of interaction (e.g., binding, regulation) and the roles each molecule plays. A key difficulty is balancing the need for high recall (finding as many potential interactions as possible) with high precision (ensuring the found interactions are accurate), as these are often competing goals. This is particularly important for applications like populating knowledge bases where accuracy is paramount, versus initial exploration where finding all possibilities is more critical.

Methods

The core of their approach is the MIRA (Multi-instance Ranking Algorithm) framework. MIRA is an online learning algorithm that allows for flexible tuning of the learning process. The authors employ a three-stage classification pipeline:

  1. Trigger Word Tagging: Identifying words in the text that signal a potential molecular event (e.g., "phosphorylates," "binds"). They experimented with various sequence tagging models, ultimately settling on a CRF-based approach enhanced with a gazetteer of trigger words derived from the training data and WordNet synsets to boost recall.
  2. Simple Event Extraction: Generating binary relations between a trigger word and a protein.
  3. Complex Event Extraction: Identifying more intricate interactions involving multiple proteins and/or simple events.

Crucially, they utilize MIRA's ability to tune the loss function during training. By adjusting how the algorithm penalizes errors, they can prioritize either precision or recall. For instance, a "high-recall" loss function heavily penalizes false negatives (missed events), while a "high-precision" loss function penalizes false positives (incorrectly identified events) more severely. They also explored a "0-1 loss" which treats all errors equally.

Datasets

The paper describes experiments conducted on data provided for the BioNLP'09 Shared Task 1. They also mention using the Genia corpora for training tokenization and sentence splitting models, and the WordNet lexical database to augment their trigger word gazetteer. No new datasets were introduced.

Key Insights and Ideas

  • Tunable Precision-Recall is Essential: The ability to dynamically adjust the trade-off between precision and recall is not just a theoretical nicety but a practical necessity for different downstream applications of event extraction.
  • Loss Function Engineering is Key: The MIRA framework's strength lies in its flexible loss functions. By carefully designing these functions (e.g., high-recall vs. high-precision), they could steer the system towards desired performance characteristics.
  • Trigger Word Identification is Foundational: The accuracy of the initial trigger word tagging significantly impacts the downstream event extraction stages. They found that treating trigger words as a distinct entity type, regardless of event type, improved recall.
  • Feature Engineering Matters: While MIRA provides a flexible learning framework, the choice of features for each stage is critical. They found that only a subset of commonly used features were truly effective for their task.

Key Results

The paper reports results for three systems tuned for different objectives:

  • High Recall System: Achieved a recall of 28.88%.
  • High Precision System: Achieved a precision of 65.58%.
  • F1-measure Optimized System: Achieved an F1-score of 33.57%.

Table 4 provides a more detailed breakdown of results for different loss functions. For example, the "High Recall" loss function achieved 39.08% recall and 48.23% precision, while the "High Precision" loss function achieved 25.86% recall and 86.54% precision. The authors note that their intended submission, if not for an error, would have placed them second on precision and sixth on recall in the competition.

Limitations and Future Work

The authors acknowledge that their official submission contained an error, leading to lower-than-expected scores. They also mention that their method for forming n-ary relations from binary ones was somewhat "ad-hoc."

For future work, they plan to:

  • Integrate Domain Knowledge: This is expected to improve recall for complex events and boost overall precision.
  • Transform Output to RDF: They intend to convert the system's output into Resource Description Framework (RDF) format and integrate it into the PIKB repository.
  • Build a Middle Ontology: They see value in creating an ontology to bridge the BioNLP and PIKB data models, a direction they plan to pursue.

Summary updated

BibTeX
@inproceedings{georgiev2009tunable,
  author = {Georgi Georgiev and Kuzman Ganchev and Vassil Momtchev and Deyan Peychev and Preslav Nakov and Angus Roberts},
  title = {{Tunable Domain-Independent Event Extraction in the MIRA Framework}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2009},
  url = {https://openreview.net/forum?id=q9aqmgdamp},
}
Figure