Abstract: Waleed Ammar, Matthew E. Peters, Chandra Bhagavatula, Russell Power. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).
2017.
Introduction
Scientific progress is often documented in millions of PDF manuscripts, making the extraction of actionable knowledge a significant challenge. Information Extraction (IE) systems are required to identify key entities—such as "nitrogen" or "electron spin"—and the relationships between them (e.g., "nitrogen is a type of material"). The ScienceIE task (SemEval-2017 Task 10) presented a complex scenario where scientific text is dense with jargon, and entities often overlap or exhibit long-distance relationships. The AI2 team addressed this by developing a robust end-to-end system that bridges the gap between entity recognition and relation extraction, which were previously often handled as isolated steps.
Methodology
The system builds upon the end-to-end neural architecture proposed by Miwa and Bansal (2016), introducing several enhancements to handle the specialized vocabulary of science.
- Entity Model: This model utilizes a stacked architecture. Token representations combine GloVe word embeddings with a character-level CNN to manage out-of-vocabulary technical terms. These are processed through Bidirectional LSTMs and a Conditional Random Field (CRF) layer using the BILOU tagging scheme.
- Semi-Supervised Learning: A critical component is the inclusion of a Neural Language Model (NLM). By pre-training on massive unlabeled corpora, the model learns the linguistic context of scientific language before supervised training.
- Relation Model: After entity identification, the system analyzes the "path" between entities. It considers both the Syntactic Path (shortest path in a dependency parse tree) and the Sequential Path (the word sequence between entities).
- External Knowledge: Gazetteers from Wikipedia and Freebase were integrated to provide the model with a "cheat sheet" of known scientific terms.
- Ensembling: The final submission employed an ensemble of 15 models to improve stability and performance.
Experimental Setup
The primary evaluation was conducted on the ScienceIE (SemEval-2017 Task 10) dataset, which contains scientific paragraphs labeled with entities (Task, Material, Process) and relations (Hyponym-of, Synonym-of). The 1B Word Benchmark was used for NLM pre-training, and external knowledge bases provided gazetteer features.
Results and Discussion
The AI2 system demonstrated state-of-the-art performance across multiple scenarios. In the end-to-end extraction task, the system ranked first.
Ablation studies revealed that the bidirectional language model provided a substantial +4.2 point boost in F1 score (from 49.9 to 54.1 on development entities), underscoring the importance of general linguistic context in specialized domains. Furthermore, the use of gazetteers and sophisticated path representations allowed the model to capture both formal grammar and informal word-order clues. The system also effectively handled overlapping entities by assigning separate labels to every token for each entity type.
Conclusion
The AI2 system's success at SemEval-2017 Task 10 highlights the efficacy of semi-supervised learning and multi-source feature integration for scientific IE. Future work may explore joint training of entity and relation models to allow for mutual feedback during the learning process, as well as more robust tokenization methods for mathematical and chemical notation.
Summary updated