TL;DR
This paper presents a novel approach for automatically extracting structured knowledge about scientific contributions from research papers. By combining BERT-based sentence classification with dependency parsing and rule-based extraction, the system aims to build a comprehensive knowledge graph of scientific advancements, making it easier for researchers to navigate and understand the ever-growing body of literature.
What Problem Does This Paper Address?
The scientific community is drowning in a sea of published papers, making it increasingly difficult for researchers to keep up with the latest advancements and identify relevant work. Manually sifting through countless articles to pinpoint key contributions is a time-consuming and often overwhelming task. The Open Research Knowledge Graph (ORKG) initiative aims to alleviate this by building a structured knowledge graph of scientific contributions, but automating this process is a significant challenge. This paper tackles the SemEval-2021 Task 11, which focuses on automatically extracting these contributions from scientific papers.
Methods
The authors developed a multi-stage approach to tackle the three sub-tasks of the NLP Contribution Graph Challenge:
- Contributing Sentence Identification (Subtask 1): They fine-tuned a SciBERT model, a BERT variant pre-trained on scientific literature, to classify sentences as either contributing to the paper's core findings or not. To address data imbalance, they employed a bi-gram filtering technique to remove a significant portion of non-contributing sentences before feeding them to the BERT model.
- Scientific Term and Predicate Phrase Extraction (Subtask 2): For this phase, they moved away from machine learning and developed a rule-based system. This system leverages dependency parsing to generate parse trees for the identified contributing sentences. They then applied a set of handcrafted rules to extract relevant noun phrases, verb phrases, and other key components, forming scientific terms and predicate phrases.
- Triple Extraction and Information Unit Classification (Subtask 3): To extract structured triples (subject, predicate, object) and classify them into predefined information units (like Research Problem, Approach, Results, etc.), they employed a CNN-based neural architecture. This model takes the extracted phrases and sentence context to categorize them into one of twelve information units.
Datasets
The primary dataset used for this task was released by D'Souza et al. (2021) for the SemEval-2021 Task 11. This dataset comprises 55,084 sentences from 236 annotated papers across 24 NLP sub-fields. Of these, 5,084 sentences are labeled as "contributing sentences," and the rest are "non-contributing." The triples are categorized into a minimum of 3 and up to 12 information units, including Research Problem, Approach, Model, Code, Dataset, Experimental Setup, Hyperparameters, Baselines, Results, Tasks, Experiments, and Ablation Analysis.
Key Insights and Ideas
- SciBERT's Efficacy: Fine-tuning a SciBERT model proved effective for identifying contributing sentences, highlighting the benefit of using models pre-trained on domain-specific text.
- Bi-gram Filtering for Imbalance: The bi-gram filtering approach was a clever way to mitigate the class imbalance issue in sentence classification, significantly improving the model's performance by reducing noise.
- Rule-Based Extraction for Precision: While BERT excels at classification, the authors found that a rule-based approach, informed by dependency parsing, was more effective for precise extraction of scientific terms and predicate phrases, especially given the limited training data for this specific sub-task.
- CNNs for Information Unit Classification: The CNN architecture provided a robust way to classify extracted information into predefined categories, enabling the structuring of extracted knowledge.
Key Results
The authors achieved respectable rankings in the competition:
- 5th rank in the overall evaluation phases.
- 5th rank in the "Sentences" evaluation phase.
- 7th rank in the "Phrases span only" evaluation phase.
While specific F1 scores for each sub-task are provided in Table 5, the paper highlights that their approach achieved an average F1 score of 0.3205 for Phase-1 (end-to-end pipeline), 0.5252 for Phase-2, Part-1 (phrases and triples extraction), and 0.5971 for Phase-2, Part-2 (triples extraction). They note that while their system performed well in identifying contributing sentences and phrase extraction, the triples extraction phase was a weaker point.
Limitations and Future Work
The authors acknowledge that their rule-based system for phrase and triple extraction, while effective, struggles with unique or highly varied sentence structures. They also note that their model sometimes misclassifies general informative sentences as contributing, especially if they contain many scientific terms.
For future work, the authors plan to focus on improving the triples extraction phase, which they identified as a key area for enhancement. They also aim to further refine the system to better capture the nuances of scientific language and extract more accurate and comprehensive knowledge graphs.
Summary updated