SemEval-2021 Task 11: NLPContributionGraph - Structuring Scholarly NLP Contributions for a Research Knowledge Graph

Jennifer D’Souza, Sören Auer, Ted PedersenACL 2021

TL;DR

This paper introduces NLPCONTRIBUTIONGRAPH (NCG), a novel shared task and dataset designed to automatically structure scholarly NLP contributions into a knowledge graph. The goal is to bridge the gap between unstructured research papers and structured semantic content, enabling more intelligent search and discovery of scientific knowledge.

What Problem Does This Paper Address?

The ever-increasing volume of scientific publications makes it challenging for researchers to keep up with advancements, even within their own narrow disciplines. Traditional search methods struggle to capture the nuanced contributions of research papers. This paper addresses this by proposing a system to automatically extract and structure these contributions into a knowledge graph (KG), making scholarly information more accessible and searchable. Existing initiatives like Google Scholar and Semantic Scholar build KGs, but NCG focuses specifically on the contributions within NLP papers, aiming for a more granular and semantically rich representation.

Methods

The NLPCONTRIBUTIONGRAPH task involved three main stages:

  1. Contribution Sentence Classification: Identifying sentences that describe the core contributions of a paper.
  2. Scientific Terms and Relations Extraction: Extracting key scientific terms and the relationships between them from these contribution sentences.
  3. Triples Extraction: Organizing the extracted terms and relations into structured (subject, predicate, object) triples to build a knowledge graph.

The task also defined 12 "Information Units" (IUs) to categorize these contributions, such as "RESEARCHPROBLEM," "APPROACH," and "RESULTS." Participants were expected to build an end-to-end pipeline for this process.

Datasets

The paper introduces the NLPCONTRIBUTIONGRAPH (NCG) dataset, which was used for the SemEval-2021 Task 11.

  • Trial Data: 50 articles across five NLP tasks.
  • Training Data: 237 articles spanning 24 tasks.
  • Test Data: 155 articles across 10 tasks. The dataset was annotated at three levels of granularity: contribution sentences, scientific terms and relations, and triples for KG building. This dataset is publicly released.

Key Insights and Ideas

  • Granularity Matters: The annotation and extraction process becomes significantly more challenging as the granularity increases, moving from sentences to phrases to triples. This is reflected in the decreasing F1 scores across these stages.
  • Pipeline vs. Subtask Performance: While end-to-end pipeline performance was relatively low, individual subtasks (like sentence classification or term extraction) achieved much higher scores when evaluated with gold-standard data from previous stages. This highlights the difficulty of integrating multiple complex NLP tasks.
  • Feature Engineering Can Be Powerful: Incorporating features like section headers and sentence position significantly improved contribution sentence classification, suggesting that leveraging structural information within papers is crucial.
  • Hybrid Approaches Excel: Systems that combined machine learning with rule-based or heuristic approaches tended to perform better, especially in the more challenging triples extraction phase.

Key Results

The best end-to-end system achieved the following F1 scores:

  • Contribution Sentence Classification: 57.27%
  • Phrases (Scientific Terms and Relations): 46.41%
  • Triples (KG Building): 22.28%

While the absolute performance for triples extraction remains low, the paper notes that subtask-specific evaluations (e.g., extracting triples given gold sentences and terms) showed much higher scores, reaching up to 61.29% F1. The CODE Information Unit was the easiest to extract, while EXPERIMENTS and TASKS were among the most complex.

Limitations and Future Work

  • Single Annotator: The dataset was annotated by a single annotator, which could introduce biases. Future work suggests a blind, multi-stage, multi-annotator process to improve robustness.
  • Non-uniform Distribution of Articles: The training data had a highly uneven distribution of articles across tasks, potentially hindering the learning of generalizable patterns. Future work aims for a more uniform representation.
  • Challenging Task: The low end-to-end performance indicates the inherent difficulty of automatically structuring scholarly contributions into a KG. Future work will focus on improving performance in each subtask and exploring more effective hybrid approaches.

Summary updated

BibTeX
@inproceedings{dsouza2021semeval,
  author = {Jennifer D’Souza and Sören Auer and Ted Pedersen},
  title = {{SemEval-2021 Task 11: NLPContributionGraph - Structuring Scholarly NLP Contributions for a Research Knowledge Graph}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure