YNU-HPCC at SemEval-2021 Task 11: Using a BERT Model to Extract Contributions from NLP Scholarly Articles

Xinge Ma, Jin Wang, Xuejie ZhangACL 2021

TL;DR

This paper presents a BERT-based system for automatically extracting research contributions from NLP scholarly articles, transforming unstructured text into structured knowledge graph triples. The system achieved competitive results in the SemEval-2021 Task 11, demonstrating the effectiveness of pre-trained language models for this complex information extraction task.

What Problem Does This Paper Address?

As the volume of scientific literature explodes, researchers face a "deluge" of information, making it challenging to keep up with advancements even within their own narrow fields. Traditional digital libraries often lack effective ways to represent and navigate this knowledge. The Open Research Knowledge Graph (ORKG) aims to address this by providing a structured representation of scholarly contributions, allowing for better tracking of research progress and interconnections between papers. However, manually creating these knowledge graphs is a monumental task. This paper tackles the challenge of automating this process by extracting structured contribution information from NLP scholarly articles.

Methods

The authors developed a multi-stage system built upon the pre-trained BERT (Bidirectional Encoder Representations from Transformers) model. The process is broken down into three subtasks, mirroring the SemEval-2021 Task 11:

  1. Contribution Sentence Identification (Subtask 1): This is framed as a binary classification problem. Each sentence in an NLP article is passed through BERT to generate word embeddings. These embeddings are then fed into a fully connected layer and a softmax function to predict whether the sentence represents a research contribution.
  2. Scientific Term and Predicate Phrase Identification (Subtask 2): This is treated as a sequence labeling task. Contribution sentences identified in Subtask 1 are processed by BERT to obtain token embeddings. These are then classified using a fully connected layer and softmax to identify tokens belonging to scientific terms or predicate phrase fragments.
  3. Triple Extraction (Subtask 3): This is the most complex stage, involving two steps:
  4. Information Unit Classification: Scientific terms and predicate phrases are classified into one of twelve predefined information units (e.g., ResearchProblem, Approach, Model). This is a multiclass-sequence classification problem.
  5. Subject-Predicate-Object Triple Formation: This is tackled using multi-label classification. Each identified phrase is tagged as a subject, predicate, or object. This is further broken down into three binary classification problems, one for each role. The system then iteratively identifies triples by finding phrases that conform to the subject-predicate-object order.

The core innovation lies in leveraging BERT's powerful contextual embeddings to handle the nuances of natural language in scholarly text for each of these stages.

Datasets

The paper utilizes the NLPContributionGraph dataset, specifically designed for SemEval-2021 Task 11. This dataset comprises NLP scholarly articles with annotated contributions. The annotations are structured into:

  • Contribution sentences.
  • Scientific terms and predicate phrases extracted from these sentences.
  • Subject-predicate-object triples representing the structured contributions.

The dataset was divided into a training set (237 articles) and a testing set (155 articles) for the final evaluation phases.

Key Insights and Ideas

  • BERT's Versatility: The paper strongly advocates for BERT's effectiveness across diverse NLP tasks, including classification, sequence labeling, and relation extraction, by fine-tuning it for each subtask.
  • Decomposition for Complexity: Breaking down the complex task of knowledge graph construction into sequential, manageable subtasks (sentence identification, phrase extraction, triple formation) proved to be a successful strategy.
  • Structured Output for Knowledge Graphs: The ultimate goal of generating (subject, predicate, object) triples is crucial for building knowledge graphs, enabling researchers to navigate and understand scientific contributions more effectively.

Key Results

The YNU-HPCC system achieved competitive performance in the SemEval-2021 Task 11:

  • Evaluation Phase 2, Part 1 (Phrases Extraction): Achieved an average F₁-score of 0.4562, ranking 7th among participating systems.
  • Evaluation Phase 2, Part 2 (Triples Extraction): Achieved an average F₁-score of 0.6541, also ranking 7th.

The paper also reports F₁-scores for individual subtasks on the development set: Subtask 1 (0.6423), Subtask 2 (0.4768), and Subtask 3 (0.4385).

Limitations and Future Work

The authors acknowledge that their system's performance, while competitive, still has significant room for improvement compared to the top-ranked systems. They suggest that future research should focus on generalizing models with better capabilities to achieve even better results. This could involve exploring more advanced BERT variants, different attention mechanisms, or more sophisticated methods for handling the complex relationships within scholarly text.

Summary updated

BibTeX
@inproceedings{ma2021ynu,
  author = {Xinge Ma and Jin Wang and Xuejie Zhang},
  title = {{YNU-HPCC at SemEval-2021 Task 11: Using a BERT Model to Extract Contributions from NLP Scholarly Articles}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure