The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction

Waleed Ammar, Matthew E. Peters, Chandra Bhagavatula, Russell PowerSemEval @ ACL 2017 · OpenReview

Abstract: Waleed Ammar, Matthew E. Peters, Chandra Bhagavatula, Russell Power. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017).

2017.

Introduction

Scientific progress is often documented in millions of PDF manuscripts, making the extraction of actionable knowledge a significant challenge. Information Extraction (IE) systems are required to identify key entities—such as "nitrogen" or "electron spin"—and the relationships between them (e.g., "nitrogen is a type of material"). The ScienceIE task (SemEval-2017 Task 10) presented a complex scenario where scientific text is dense with jargon, and entities often overlap or exhibit long-distance relationships. The AI2 team addressed this by developing a robust end-to-end system that bridges the gap between entity recognition and relation extraction, which were previously often handled as isolated steps.

Methodology

The system builds upon the end-to-end neural architecture proposed by Miwa and Bansal (2016), introducing several enhancements to handle the specialized vocabulary of science.

The end-to-end neural architecture for entity and relation extraction, incorporating a neural language model and gazetteer features.
  1. Entity Model: This model utilizes a stacked architecture. Token representations combine GloVe word embeddings with a character-level CNN to manage out-of-vocabulary technical terms. These are processed through Bidirectional LSTMs and a Conditional Random Field (CRF) layer using the BILOU tagging scheme.
  2. Semi-Supervised Learning: A critical component is the inclusion of a Neural Language Model (NLM). By pre-training on massive unlabeled corpora, the model learns the linguistic context of scientific language before supervised training.
  3. Relation Model: After entity identification, the system analyzes the "path" between entities. It considers both the Syntactic Path (shortest path in a dependency parse tree) and the Sequential Path (the word sequence between entities).
  4. External Knowledge: Gazetteers from Wikipedia and Freebase were integrated to provide the model with a "cheat sheet" of known scientific terms.
  5. Ensembling: The final submission employed an ensemble of 15 models to improve stability and performance.

Experimental Setup

The primary evaluation was conducted on the ScienceIE (SemEval-2017 Task 10) dataset, which contains scientific paragraphs labeled with entities (Task, Material, Process) and relations (Hyponym-of, Synonym-of). The 1B Word Benchmark was used for NLM pre-training, and external knowledge bases provided gazetteer features.

Results and Discussion

The AI2 system demonstrated state-of-the-art performance across multiple scenarios. In the end-to-end extraction task, the system ranked first.

Comparison of the AI2 system (Ours) against other top-performing teams in the ScienceIE task.

Ablation studies revealed that the bidirectional language model provided a substantial +4.2 point boost in F1 score (from 49.9 to 54.1 on development entities), underscoring the importance of general linguistic context in specialized domains. Furthermore, the use of gazetteers and sophisticated path representations allowed the model to capture both formal grammar and informal word-order clues. The system also effectively handled overlapping entities by assigning separate labels to every token for each entity type.

Conclusion

The AI2 system's success at SemEval-2017 Task 10 highlights the efficacy of semi-supervised learning and multi-source feature integration for scientific IE. Future work may explore joint training of entity and relation models to allow for mutual feedback during the learning process, as well as more robust tokenization methods for mathematical and chemical notation.

Summary updated

BibTeX
@inproceedings{ammar2017ai,
  author = {Waleed Ammar and Matthew E. Peters and Chandra Bhagavatula and Russell Power},
  title = {{The AI2 system at SemEval-2017 Task 10 (ScienceIE): semi-supervised end-to-end entity and relation extraction}},
  booktitle = {SemEval @ ACL},
  year = {2017},
  url = {https://openreview.net/forum?id=QxXfvA05JO},
}
Figure