Constructing large scale biomedical knowledge bases from scratch with rapid annotation of interpretable patterns

Julien Fauqueur, Ashok Thillaisundaram, Theodosia TogiaACL 2019

TL;DR

This paper introduces a novel method for automatically constructing large-scale biomedical knowledge bases by extracting interpretable patterns from text. It allows domain experts to efficiently annotate vast amounts of data by focusing on pattern meaning rather than individual sentence examples, significantly speeding up knowledge base creation, especially when existing data is scarce.

What Problem Does This Paper Address?

Building comprehensive biomedical knowledge bases is crucial for applications like drug discovery. However, creating these bases is challenging due to the scarcity of specific, task-relevant facts in existing literature. Manually extracting these facts or training complex models requires extensive expert time and often relies on laborious, ad-hoc rules that are difficult to reuse. Existing automated methods either still require significant human effort or produce "black-box" rules that are hard to interpret and integrate into expert workflows. This paper aims to bridge this gap by enabling rapid, expert-guided construction of knowledge bases using interpretable patterns.

Methods

The core of the proposed method lies in extracting and presenting "interpretable patterns" from sentences that contain pairs of entities (e.g., gene-disease). These patterns are essentially text-like representations of linguistic structures that indicate a potential relationship. The system generates various types of patterns, including:

  • KEYWORDS: Specific words or lemmas within the sentence.
  • PATH: The shortest dependency path between two entities.
  • SENTENCE_ROOT: The root of the sentence's dependency graph.
  • PATH_BETWEEN_ROOTS: The path connecting the sentence root and the path root.
  • SENTENCE_ROOT_DESCENDANTS: Direct descendants of the sentence root.
  • PATH_ROOT_DESCENDANTS: Direct descendants of the path root.

These patterns are then "lexicalized" into human-readable "simplifications." The system offers several workflows:

  1. "No expert - no labels": Extracts new pairs based on high-frequency patterns.
  2. "Expert - no labels": Presents top-ranked simplifications to a domain expert for annotation ("Yes," "No," "Maybe"). This allows experts to indirectly annotate thousands of candidate pairs by evaluating patterns.
  3. "No expert but labels": Uses existing gold-standard data to rank simplifications based on precision and recall.
  4. "Expert with labels": Combines expert annotation with labeled data, using metrics to prioritize impactful simplifications.

The system also incorporates filtering for negation and hedging to ensure extracted facts are affirmative. Additionally, it uses clustering to group similar simplifications, improving annotation efficiency and recall.

Datasets

The experiments utilize two primary biomedical datasets:

  • DisGeNET: A comprehensive platform integrating information on human disease-associated genes and variants.
  • Comparative Toxicogenomics Database (CTD): Contains relation types like 'marker/mechanism' and 'therapeutic'.

The CTD dataset was further refined into "CTD therapeutic" for specific experiments, focusing on therapeutic gene-disease pairs. The corpus for pattern extraction consists of sentences from PubMed articles that contain exactly one gene-disease pair.

Key Insights and Ideas

  • Interpretable Patterns as Bridges: The key innovation is using interpretable patterns (simplifications) as a bridge between raw text and expert knowledge. Experts can quickly grasp the meaning of a pattern and decide its validity, rather than sifting through countless individual sentences.
  • Batch Annotation Power: By annotating patterns, experts can implicitly annotate thousands of entity pairs, drastically reducing annotation time and effort.
  • Scalability from Scarcity: The method is particularly effective when little to no labeled data is available, demonstrating that knowledge base construction can be initiated from scratch with expert guidance.
  • Trade-off Management: The paper highlights the inherent trade-off between pattern expressiveness (specificity) and coverage (breadth). More expressive patterns are less likely to subsume many sentences, requiring careful selection.
  • Leveraging Weak Supervision: The "no expert but labels" workflow shows how existing, potentially noisy, labeled data can be used to automatically rank and select patterns, even without direct expert input.

Key Results

  • Rapid Knowledge Extraction: The "expert - no labels" workflow, with an expert annotating 200 simplifications and 20 sentences each, took about 3 hours and resulted in discovering thousands of high-quality gene-disease therapeutic pairs.
  • High Precision and Recall: The proposed workflows achieved significant results in intrinsic evaluation. For instance, the "no expert but labels" workflow on the CTD therapeutic dataset yielded 30,006 new pairs with a precision of 0.894 and F-score of 0.772.
  • Improved Knowledge Base Completion: Extrinsic evaluation showed that augmenting a knowledge base completion model with pairs discovered by the automated workflow led to higher mean average precision (MAP) compared to using only the initial seed dataset.
  • Expert-Guided Refinement: Incorporating expert feedback ("expert with labels") generally led to higher precision at the cost of some recall, demonstrating the ability to control the quality of extracted pairs.

Limitations and Future Work

  • Sentence-Level Focus: The current system primarily focuses on sentences containing exactly one gene-disease pair. Extending it to handle multiple pairs within a sentence is a straightforward next step.
  • Binary Relations: The current work focuses on binary relationships. Future work could explore handling higher-order (ternary) relations by analyzing dependency subgraphs connecting more than two entities.
  • Negative Examples: The system primarily discovers positive examples. Discovering explicit negative examples is a potential area for future research.
  • Continuous Annotation: The paper suggests exploring continuous annotation schemes (e.g., percentage of correct examples) as part of a more sophisticated active learning strategy.
  • Semantic Representations: Future work could involve extracting patterns from semantic representations or mapping patterns to vector spaces for clustering and better expressive power/coverage balance.

Summary updated

BibTeX
@inproceedings{fauqueur2019constructing,
  author = {Julien Fauqueur and Ashok Thillaisundaram and Theodosia Togia},
  title = {{Constructing large scale biomedical knowledge bases from scratch with rapid annotation of interpretable patterns}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
}
Figure