Duluth at SemEval-2021 Task 11: Applying DeBERTa to Contributing Sentence Selection and Dependency Parsing for Entity Extraction

Anna Martin, Ted PedersenACL 2021

TL;DR

This paper presents the Duluth system for SemEval-2021 Task 11, which automates the extraction of scientific contributions from NLP articles. The system leverages DeBERTa for sentence classification and dependency parsing to identify entities and relations, structuring them into subject-predicate-object triples. While achieving competitive rankings, the work highlights challenges in handling imbalanced datasets and the nuances of scientific language.

What Problem Does This Paper Address?

The ever-increasing volume of scientific literature makes it challenging for researchers to keep up with the latest advancements, even within their own fields. Manually extracting and structuring key contributions from papers is a time-consuming and laborious task. SemEval-2021 Task 11 aimed to address this by automating the annotation of scientific papers to build a scholarly contributions graph. This involves identifying sentences that describe contributions, extracting scientific entities and their relationships, and organizing these into structured triples that represent the type of contribution.

Methods

The Duluth system employs a multi-stage approach:

  • Contribution Sentence Selection: The core of this stage uses DeBERTa, a powerful pre-trained language model, fine-tuned for multi-class sentence classification. Instead of a simple binary classification (contributing/non-contributing), the model classifies sentences into one of twelve predefined "information units" that describe the nature of the contribution. This approach aims to capture more nuanced information early in the pipeline.
  • Entity and Relation Extraction:
    • Dependency Parsing: Stanford Core NLP's dependency parser is used to analyze the structure of the selected contribution sentences. This helps identify the root verb phrase, noun subject phrases, and dependent object phrases.
  • Entity Tagging: A Maximum Entropy Markov Model (MEMM) is trained to identify likely scientific entities within the extracted noun phrases. This model utilizes features like word type, part-of-speech tags, and word shape.
  • Triple Formation: The system extracts subject-predicate-object (SPO) triples by combining the parsed dependency structures and the identified scientific entities. The verb phrase is typically identified as the predicate, with noun phrases serving as subjects and objects.

Datasets

The paper utilizes datasets provided by the SemEval-2021 Task 11 organizers. These include:

  • Training Data: Used for fine-tuning the DeBERTa model and training the MEMM entity tagger.
  • Trial Dataset: Used for initial validation and experimentation before the first evaluation phase.
  • Test Data: Used for evaluating the system's performance during the official competition phases.
  • Gold Data: Provided after each phase, containing the correct annotations (sentences, entities, triples) for evaluation.

The paper notes that the training data exhibits a significant class imbalance, with over 90% of sentences being non-contributing.

Key Insights and Ideas

  • DeBERTa's Power for Nuanced Classification: Fine-tuning DeBERTa for multi-class classification of contribution types (information units) proved more effective than a simple binary classification, allowing for richer semantic understanding early in the pipeline.
  • Dependency Parsing as a Foundation: Leveraging dependency parses provided a robust way to identify the grammatical structure of sentences, enabling the extraction of subject-predicate-object relationships.
  • Challenges of Imbalanced Data: The extreme imbalance in the training data (over 90% non-contributing sentences) significantly impacted the performance of simpler classifiers like logistic regression and decision trees, highlighting the need for more sophisticated models like DeBERTa.
  • The Nuance of "Scientific Entities": The definition of "scientific entities" proved challenging, as the system struggled with common nouns that might be relevant in a scientific context but weren't strictly technical terms. This led to filtering issues that negatively impacted performance.
  • Interconnectedness of Contributions: The task of forming triples revealed that contributions often build on each other, requiring careful handling of overlapping entities and predicates to accurately represent these relationships.

Key Results

The Duluth system achieved competitive rankings in the SemEval-2021 Task 11:

  • Phase 1 (End-to-End Pipeline): Ranked 5th out of 7.
  • Phase 2 Part 1 (Phrases and Triples): Ranked 6th out of 8.
  • Phase 2 Part 2 (Triples Extraction): Ranked 5th out of 8.

Specific F1 scores are provided:

  • Sentence Selection (Binary Classification): 0.38095
  • Sentence Selection (Multi-class Information Units): 0.6441
  • Triples Extraction (Phase 2 Part 2): 0.2762
  • Information Units (Triples Extraction): 0.7556

The paper also details confusion matrices and individual F1 scores for each information unit, revealing specific areas of confusion (e.g., EXPERIMENTAL SETUP vs. HYPERPARAMETERS).

Limitations and Future Work

  • Sentence-Level Information: The current sentence selection relies solely on sentence-level information, missing valuable context from document headers or surrounding sentences.
  • Lack of Feedback Loop: The system lacks a mechanism for the later stages (entity/triple extraction) to provide feedback to the earlier sentence selection stage.
  • Entity Extraction Approach: The reliance on a grammatical approach for entity extraction might not fully capture the nuances of scientific language, suggesting a potential benefit from neural entity classification models.
  • Future Directions:
    • Document-Level Classification: Investigating document-level classification approaches to leverage broader context.
  • Neural Entity Classification: Incorporating neural models for entity and triple extraction to improve accuracy.
  • Feedback Mechanisms: Developing methods for later pipeline stages to influence earlier ones.
  • Handling Pronouns: Improving the handling of pronouns referring to entities from previous sentences.

Summary updated

BibTeX
@inproceedings{martin2021duluth,
  author = {Anna Martin and Ted Pedersen},
  title = {{Duluth at SemEval-2021 Task 11: Applying DeBERTa to Contributing Sentence Selection and Dependency Parsing for Entity Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure