Open Information Extraction on Scientific Text: An Evaluation

Paul Groth, Mike Lauruhn, Antony Scerri, Ron Daniel Jr.COLING 2018

TL;DR

This paper evaluates the performance of Open Information Extraction (OIE) systems on scientific text, finding they perform significantly worse than on general text. The authors introduce a new evaluation methodology using crowd-sourcing and provide an error analysis to guide future improvements in extracting structured information from scientific literature.

What Problem Does This Paper Address?

The scientific literature is exploding, making it increasingly difficult to extract valuable information. Open Information Extraction (OIE) aims to automatically create structured data from text, which is crucial for tasks like building knowledge bases and answering questions. However, existing OIE systems have primarily been evaluated on general text like news articles or Wikipedia. This leaves a significant gap: how well do these systems actually perform on the complex and specialized language found in scientific publications? This paper addresses this by providing the first systematic evaluation of state-of-the-art OIE systems specifically on scientific text.

Methods

The authors evaluated two prominent OIE systems: OpenIE 4 and MinIE. Instead of relying on pre-defined "gold standard" extractions, they adopted a novel crowd-sourcing approach. Sentences and their extracted triples were presented to Amazon Mechanical Turk workers, who judged the correctness of each triple. This method allows for a larger number of annotators and focuses on the fundamental question: is the extracted information actually correct? They also adapted existing labeling instructions for crowd workers and implemented a qualification process to ensure annotator quality.

Datasets

The evaluation used two datasets:

  • WIKI: A set of 200 Wikipedia sentences, previously used in other OIE evaluations, allowing for rough comparisons.
  • SCI: A new dataset of 220 sentences sourced from the Open Access Scientific and Technical Literature (OA-STM) corpus, covering 10 diverse scientific disciplines (agriculture, astronomy, biology, chemistry, computer science, earth science, engineering, materials science, math, and medicine). Sentences were randomly selected from articles.

Key Insights and Ideas

  • Scientific Text is Harder: OIE systems struggle significantly with scientific text compared to general text. This suggests that the specialized vocabulary, complex sentence structures, and nuanced relationships in science pose a unique challenge.
  • Crowd-Sourcing for Evaluation: Using crowd-sourcing for OIE evaluation, rather than relying solely on expert annotators, is a viable and scalable approach. It allows for a broader range of judgments and can reveal nuances missed by smaller expert groups.
  • Error Patterns are Informative: The error analysis highlights specific challenges, such as handling pronouns, complex prepositional phrases, abbreviations, and mathematical formulas, which are common in scientific writing.

Key Results

  • Precision Drop: OIE systems showed a significant drop in precision on scientific text (0.34 P) compared to Wikipedia text (0.54 P). For OpenIE 4, precision was 0.19 higher on Wikipedia.
  • Annotator Agreement: Annotator agreement was lower on scientific text (0.72 SD 0.25) than on Wikipedia text (0.80 SD 0.24), indicating greater subjectivity or difficulty in judging correctness.
  • System Performance: OpenIE 4 generally outperformed MinIE in terms of precision (0.56 P vs. 0.39 P) and produced more compact relation types. However, MinIE extracted more correct triples overall, suggesting a trade-off between precision and recall.
  • Hypothesis Rejection: The authors rejected their first hypothesis (H1) that OIE performance is similar across scientific and general text, and their second hypothesis (H2) that performance is similar between scientific and medical content.

Limitations and Future Work

  • Limitations: The study acknowledges that the crowd-sourcing approach, while beneficial, might lead to lower agreement on complex cases. The evaluation focused on binary relations, not n-ary ones. The true recall of the systems could not be fully determined due to the evaluation methodology.
  • Future Work: The authors suggest several avenues for improvement:
  • Exploring the "Crowd Truth" framework to better handle disagreement.
  • Investigating n-ary relations and negative relations.
  • Developing better guidelines and tasks for crowd-sourcing complex analyses.
  • Expanding the evaluation dataset to include more sentences.
  • Integrating coreference resolution and other NLP components like syntactic features to improve extraction.

Summary updated

BibTeX
@inproceedings{groth2018open,
  author = {Paul Groth and Mike Lauruhn and Antony Scerri and Ron Daniel Jr.},
  title = {{Open Information Extraction on Scientific Text: An Evaluation}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure