Extracting meronyms for a biology knowledge base using distant supervision

Xiao Ling, Peter Clark, Daniel S. WeldAKBC @ CIKM 2013 · DOI

TL;DR

This paper tackles the challenge of automatically building a knowledge base of biological part-whole relationships (meronyms) from text. They introduce a novel generalization of multi-instance learning that accounts for an unknown percentage of positive examples in a training "bag," improving extraction performance.

What Problem Does This Paper Address?

Building comprehensive knowledge bases (KBs) of scientific information, like the relationships between biological entities and their parts (meronyms), is crucial for question-answering systems. However, manually encoding these facts is incredibly time-consuming and impractical. Previous attempts at automatic meronym extraction have struggled to achieve both high precision (correctly identified relationships) and high recall (finding most of the true relationships). This paper aims to improve meronym extraction by leveraging "distant supervision" and a more flexible multi-instance learning approach.

Methods

The core of the paper's contribution lies in its novel approach to distant supervision and multi-instance learning (MIL).

  • Distant Supervision: Instead of relying on manually labeled sentences, they use an existing knowledge base (a database of known meronym facts) to automatically label sentences. If a sentence mentions two entities that are known to have a part-whole relationship in the KB, it's labeled as a "positive bag." Sentences with entity pairs not found in the KB are used to form "negative bags." This process inherently introduces noise, as not all labeled sentences will accurately reflect the meronym relationship.
  • Generalized Multi-Instance Learning: Traditional MIL assumes that in a "positive bag" of instances (sentences), at least one instance is a true positive. This paper generalizes this to assume that a fixed, but unknown percentage of instances in a positive bag are true positives. This allows for more nuanced learning, as it acknowledges that multiple sentences might express the same meronym relationship. They adapt existing graphical models (specifically, extending MULTIR) to incorporate this percentage constraint and use grid search with cross-validation to find the optimal percentage for each relation.
  • Mention Detection: They improve entity mention detection by incorporating coreference resolution (identifying when pronouns or other phrases refer to the same entity) and expanding mentions to include partitive nouns (e.g., recognizing "a sequence of nucleitide bases" as referring to "nucleitide bases").
  • Negative Example Generation: To create more robust training data, they employ several strategies: using manually labeled negative examples, creating "reversed" instances (if A is part of B, then B is not part of A), and using a transitive closure of negative relations.

Datasets

  • Knowledge Base: A seed database of meronyms from the biology domain, originally created for Project Halo. This includes both context-free facts and facts with contextual qualifications.
  • Unlabeled Corpus: A digitized version of a popular biology textbook (41,892 sentences).
  • Out-of-Domain Data: WordNet, a large lexical database, was used to incorporate general meronym facts, though type information had to be removed due to incompatibility.
  • Evaluation Data: A held-out test set was created and validated by two human annotators, consisting of 172 has-part facts and 206 "not applicable" (NA) facts.

Key Insights and Ideas

  • The "at least one" assumption is too strict: For meronym extraction, it's common for multiple sentences to express the same part-whole relationship. Assuming only one is correct can lead to missed learning opportunities. The generalized percentage assumption allows the model to learn from more of the available data.
  • Coreference and partitive nouns significantly boost mention detection: Simply relying on string matching is insufficient. Incorporating linguistic cues like coreference and partitive nouns leads to more accurate identification of entities, which is foundational for relation extraction.
  • Negative example generation is crucial for precision/recall trade-off: Different strategies for generating negative examples have distinct impacts on precision and recall. The "TRANS" strategy (transitive closure) yielded the highest F1 score, suggesting a good balance.
  • Out-of-domain data offers limited gains without ontological alignment: While using general meronym facts from WordNet showed some improvement, the lack of type compatibility between WordNet and the biology ontology limited the benefits. This highlights the importance of semantic alignment for effective knowledge transfer.

Key Results

  • The proposed generalized multi-instance learning approach (MULTIR(p)) achieved a small but distinct improvement in F1 score (0.788) compared to the standard "at least one" MIL approach (MULTIR, F1 of 0.769) on the test set.
  • Accurate mention detection, incorporating coreference, improved F1 scores from 0.733 to 0.740 on the test set.
  • The "TRANS" strategy for negative example generation achieved the highest F1 score (0.769) among the tested negative example generation methods.
  • Adding out-of-domain WordNet meronyms provided only small improvements (e.g., F1 of 0.655 vs. 0.636 with p=0.2), largely due to the lack of type features.

Limitations and Future Work

  • Ontological Alignment: The authors acknowledge that the lack of type features when using out-of-domain WordNet data significantly impacted performance. Future work could focus on better ontological alignment to leverage external knowledge more effectively.
  • Feature Engineering: While they used standard "Mintz features," further exploration of domain-specific features could potentially improve performance.
  • Scalability: The paper focuses on a specific domain (biology) and a specific relation (meronym). Scaling this approach to other domains and a wider variety of relations would be a natural next step.
  • Completeness of KB: The distant supervision approach relies on an existing KB. The completeness and accuracy of this seed KB will influence the quality of the extracted data.

Summary updated

BibTeX
@inproceedings{ling2013extracting,
  author = {Xiao Ling and Peter Clark and Daniel S. Weld},
  title = {{Extracting meronyms for a biology knowledge base using distant supervision}},
  booktitle = {AKBC @ CIKM},
  year = {2013},
  doi = {10.1145/2509558.2509560},
}
Figure