Scalable, Semi-Supervised Extraction of Structured Information from Scientific Literature

Kritika Agrawal, Aakash Mittal, Vikram PudiNAACL 2019

TL;DR

This paper introduces a scalable, semi-supervised method to extract key concepts (Aim, Method, Result) from scientific literature, building a knowledge graph that summarizes research trends. This approach leverages domain-independent word embeddings and a bootstrapping framework, achieving state-of-the-art performance while being more generalizable than previous methods.

What Problem Does This Paper Address?

The ever-growing volume of scientific literature makes it increasingly difficult for researchers to keep up with advancements, identify relevant work, and understand the landscape of a research area. Traditional information extraction methods struggle with the unique terminology and semi-structured nature of scientific papers. Furthermore, annotating scientific text for supervised learning is costly and requires domain expertise. This paper tackles the challenge of automatically extracting structured information—specifically, the core contributions (Aim, Method, Result) of research papers—to build comprehensive knowledge graphs that can facilitate research navigation, trend analysis, and discovery.

Methods

The authors propose a novel, scalable, semi-supervised method for extracting "Aim," "Method," and "Result" concepts from scientific articles. Their approach combines domain-based word embeddings (specifically, BERT embeddings fine-tuned on a DBLP dataset) with a bootstrapping framework.

Here's a breakdown of their key methodological contributions:

  • Concept Definition: They define "Aim" as the paper's target, "Method" as the techniques used, and "Result" as the output or contribution.
  • Extraction Sources: Information is extracted from the Title, Abstract, and Citation Contexts of papers, as these sections are considered concise summaries and less prone to noise.
  • Semi-Supervised Learning: They start with a small set of annotated "seed features" and use a bootstrapping approach to iteratively extract more features from unlabeled data.
  • Phrase Classification: Instead of treating it as a sequence labeling task, they frame concept extraction as a multi-class classification problem. They classify noun phrases (from Title/Abstract) and named entities (from citation contexts) as Aim, Method, or Result.
  • k-Nearest Neighbors (k-NN): A k-NN classifier is used on top of BERT embeddings to capture semantic similarity between phrases.
  • Knowledge Graph Construction: Extracted concepts are integrated into a citation graph. Paper nodes are linked to entity nodes (concepts) via "Aim," "Method," and "Result" relations. To handle semantic similarity across different phrases, they employ DBSCAN clustering based on lexical similarity within conference groups, followed by merging clusters across conferences.

Datasets

  • DBLP Citation Network (version 7): This forms the basis of their large-scale dataset. They extracted full text for 465,355 papers, then pruned it to the largest connected component of the citation network, resulting in 332,793 papers and 1,508,560 citation links. This dataset was used for fine-tuning BERT embeddings and for their main experiments.
  • SemEval 2017 Task 10 dataset: This dataset, containing 500 scientific paragraphs from physics, material science, and computer science, was used for seed feature generation for TASK (related to AIM) and PROCESS (related to METHOD).
  • Gupta and Manning (2011) dataset: This dataset of titles and abstracts from 474 ACL Anthology publications, annotated with FOCUS (related to AIM) and TECHNIQUE (related to METHOD), was also used for seed feature extraction.
  • Manually Annotated Data: For the "Result" concept, they manually annotated titles and abstracts of 100 computer science publications.
  • Evaluation Dataset: They used a subset of the Gupta and Manning (2011) dataset (197 papers) for evaluating Aim and Method concepts, as Result concepts were not annotated.

Key Insights and Ideas

  • Citation Context is Rich: The text surrounding citation markers provides valuable "micro-summaries" of cited papers, making them strong candidates for concept extraction.
  • Domain Independence is Achievable: By using domain-based word embeddings and a semi-supervised approach, their method can generalize across different scientific domains, as demonstrated by using seed features from one domain to extract concepts in another.
  • Knowledge Graph for Summarization: A citation graph augmented with extracted Aim, Method, and Result concepts can effectively summarize a research community, revealing trends and relationships that might not be apparent from individual papers.
  • Clustering for Concept Merging: Grouping papers by conference before clustering phrases helps to mitigate issues where similar-sounding phrases from different domains might be incorrectly merged.

Key Results

  • Performance: Their proposed approach achieved F1 scores comparable to state-of-the-art methods for Aim and Method concepts. While their F1 scores were sometimes lower than specific prior works, their recall was consistently higher.
  • Scalability: The method demonstrated scalability, successfully processing a large multi-domain dataset (332,793 papers). They note that a prior bootstrapping algorithm struggled to achieve good scores on their multi-domain dataset.
  • Trend Analysis: They successfully applied their method to analyze research trends in Computational Linguistics and Computer Vision, showing the transition of topics from "Aim" to "Method" over time.
  • Graph Size: Their constructed graph contains 332,242 paper nodes, 362,654 entity nodes, and millions of relation instances.

Limitations and Future Work

  • Result Concept Annotation: The lack of comprehensive "Result" annotations in existing datasets limited their ability to fully evaluate this concept.
  • Precision vs. Recall Trade-off: They acknowledge that their precision might be lower because they considered all noun phrases, whereas prior work might have had fewer constraints.
  • Future Work: The paper suggests that the constructed knowledge graph can be used for various applications like academic search, recommendation systems, and trend analysis, with the specific query interfaces being out of scope for this paper. They also highlight the potential for further exploration of the graph's structure and its applications.

Summary updated

BibTeX
@inproceedings{agrawal2019scalable,
  author = {Kritika Agrawal and Aakash Mittal and Vikram Pudi},
  title = {{Scalable, Semi-Supervised Extraction of Structured Information from Scientific Literature}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure