TL;DR
This paper presents the winning system for the SemEval-2021 Task 11, which focuses on extracting structured knowledge (sentences, phrases, and triples) from scholarly NLP articles. Their approach significantly improved upon existing methods by combining BERT with novel sampling strategies and adversarial training techniques, achieving a top F1 score of 0.4703.
What Problem Does This Paper Address?
The paper tackles the challenge of automatically extracting structured knowledge from academic papers, specifically in the field of Natural Language Processing (NLP). This process, known as Information Extraction (IE), is crucial for building Knowledge Graphs (KGs) that represent scholarly contributions in a machine-readable format. The NLPContributionGraph task aims to develop systematic patterns for identifying and structuring these contributions, creating a valuable resource for researchers to manage and navigate academic literature more efficiently. The core problem is to accurately identify and extract three types of information: contribution sentences, scientific term/predicate phrases, and subject-predicate-object triples.
Methods
The winning system is built upon the BERT (Bidirectional Encoder Representations from Transformers) architecture, a powerful pre-trained language model. To boost BERT's performance, the authors employed two key strategies:
- Sampling Strategies: To address class imbalance issues (where contribution sentences are rare compared to non-contribution sentences), they used under-sampling for sentence and triple extraction. For triple extraction, they further explored Random Replacement (RR) and Random Selection (RS) to generate more diverse negative samples, finding that a combination of RR and RS yielded the best results.
- Adversarial Training: Recognizing that machine learning models can be vulnerable to small perturbations, they adopted Friendly Adversarial Training (FAT). FAT, an early-stopped version of Projected Gradient Descent (PGD), generates "friendly" adversarial data rather than the most adversarial data. This approach aims to improve the model's robustness and generalization without significantly hurting its performance, leading to better accuracy and more stable training.
The system comprises three main modules: a Sentence Extractor (using BERT as a binary classifier), a Phrases Extractor (using BERT for sequence labeling), and a Triples Extractor (which classifies candidate triples generated by a Triples Generator).
Datasets
The primary dataset used is the one provided for the SemEval-2021 Task 11: NLPContributionGraph. This dataset contains hundreds of NLP scholarly articles annotated with:
- Contribution sentences.
- Scientific term and predicate phrases.
- Subject-predicate-object triple statements for KG building.
The paper also mentions using a validation set for ablation studies and testing on five different domains (Question Answering, Relation Extraction, Sentence Classification, Sentence Compression, and Text Generation) for evaluating sentence extraction features.
Key Insights and Ideas
- Sampling is Crucial for Imbalanced Data: The paper strongly emphasizes that effective sampling strategies are vital for tasks with highly imbalanced datasets, such as identifying contribution sentences. Simple over-sampling can lead to overfitting, while carefully designed under-sampling and diverse negative sample generation (like RR and RS) significantly improve generalization.
- Adversarial Training Enhances Robustness and Generalization: Contrary to some findings in computer vision, adversarial training (specifically FAT) proved beneficial for NLP tasks, improving both the generalization and robustness of BERT models. This led to more stable training and better performance on validation sets.
- BERT is a Strong Foundation, but Needs Augmentation: While BERT provides a powerful baseline, the paper demonstrates that combining it with specialized techniques like sampling and adversarial training is essential to achieve state-of-the-art results in complex information extraction tasks.
- Task-Specific Strategies Matter: Different tasks within the NLPContributionGraph (SE, PE, TE) benefited from tailored approaches. For instance, under-sampling was used for SE and TE, while the PE task relied on ensembling learning (though it was noted that ensembling was not applied in the competition for PE, leading to a lower F1 score for that specific task).
Key Results
The system achieved a winning average F1 score of 0.4703 in the End-to-end Pipeline phase, significantly outperforming the second-placed system's average F1 of 0.3828.
- Task SE: Achieved an F1 score of 0.5619.
- Task PE: Achieved an F1 score of 0.4522.
- Task TE: Achieved an F1 score of 0.1379.
The authors note that while SE and PE performance was strong, TE could be further improved, particularly by addressing the limitation of only considering sequential triples. Adversarial training, especially FAT, consistently showed better F1 scores across all tasks compared to natural training or FGM. Adding both title and context features for sentence extraction significantly boosted accuracy, reaching up to 0.6977 for QA.
Limitations and Future Work
- Triples Extraction Limitation: The paper acknowledges that the Triples Extractor's current approach of only considering sequential triples can lead to missed information, suggesting this as an area for future improvement.
- Ensembling for PE: While ensembling was found to be effective for the Phrases Extractor, it was not applied in the competition, leading to a lower F1 score for PE. Future work could explore integrating ensembling more effectively.
- Generalizability of Adversarial Training: While FAT showed promise, further research could explore its application and effectiveness across a wider range of NLP tasks and datasets.
- Computational Cost: Adversarial training can increase computational costs, and future work might focus on more efficient adversarial training methods.
Summary updated