SEPT: Improving Scientific Named Entity Recognition with Span Representation

Xian-Ling MaoarXiv 2019 · arXiv

TL;DR

This paper introduces SEPT, a novel approach to scientific named entity recognition (NER) that improves upon existing span-based models. By strategically under-sampling negative spans and simplifying the architecture, SEPT achieves state-of-the-art results on the SCIERC dataset, demonstrating the effectiveness of balancing training data and leveraging pre-trained language models.

What Problem Does This Paper Address?

Extracting structured information from scientific literature is crucial for advancing research. Named Entity Recognition (NER) is a foundational task in this area, aiming to identify and classify key entities like genes, proteins, or chemicals within scientific texts. While span-based NER models have shown promise in handling complex entity structures (like overlapping or nested entities), they often suffer from computational inefficiency due to the need to score all possible spans in a document. Furthermore, the scarcity of annotated scientific datasets exacerbates the challenge of training robust NER models. This paper addresses the need for a more efficient and accurate span-based NER system specifically for scientific text, aiming to improve performance without relying on complex multi-task frameworks.

Methods

SEPT (Span Extractor with Pre-trained Transformers) builds upon the strengths of span-based models and pre-trained language models like SciBERT. The core innovations lie in how it handles the training data and its architecture:

  • Under-sampling Negative Spans: Instead of enumerating all possible negative spans (spans that do not represent an entity), SEPT randomly samples a subset. This significantly reduces the search space and helps balance the highly imbalanced positive (actual entities) and negative samples, a common issue in NER.
  • Simplified Span Extractor: Leveraging the inherent ability of transformer models (like SciBERT) to capture token interactions through self-attention, SEPT simplifies the span extractor. It uses a straightforward max-pooling operation over token embeddings within a span to derive its representation, eliminating the need for explicit features like endpoint features, content attention, or span length embeddings that were used in prior span-based models.
  • Pre-trained SciBERT: The model utilizes SciBERT, a BERT model pre-trained on a large corpus of scientific text, to obtain rich contextual embeddings for each token. This is crucial for understanding the nuances of scientific language.
  • MLP Classifier: A simple Multi-Layer Perceptron (MLP) is used to classify the extracted span representations into different entity types.

Datasets

The primary dataset used for training and evaluation in this paper is SCIERC. The authors note that SCIERC is the only computer science-related fine-grained information extraction dataset available, highlighting its importance for their research.

Key Insights and Ideas

  • The Power of Under-sampling: The paper demonstrates that strategically under-sampling negative spans is a simple yet effective way to improve both the performance and efficiency of span-based NER models, especially when dealing with imbalanced datasets.
  • Transformer Embeddings Simplify Feature Engineering: The success of SEPT shows that the rich contextual information captured by pre-trained transformer models like SciBERT can make explicit, hand-crafted features for span representation redundant. This simplifies the model architecture and potentially improves generalization.
  • Single-Task Focus Can Be Sufficient: While multi-task learning has been beneficial for information extraction, this work suggests that significant improvements can be achieved by focusing on optimizing a single task (NER) with a well-designed model and training strategy.

Key Results

SEPT achieves state-of-the-art results on the SCIERC dataset. Specifically, it reports an F1 score of 0.677, outperforming previous transformer-based systems. The paper also highlights that even with a simplified architecture, SEPT matches the performance of more complex span extractors. Experiments on the number of negative samples show that performance degrades when too many negative samples are used, with an optimal point around 250 negative samples.

Limitations and Future Work

The authors acknowledge that their current work focuses solely on named entity recognition. They suggest that in future work, they will investigate whether the SEPT model can be jointly trained with relation extraction and other metadata from scientific papers. This would allow for a more comprehensive information extraction system.

Summary updated

Abstract

We introduce a new scientific named entity recognizer called SEPT, which stands for Span Extractor with Pre-trained Transformers. In recent papers, span extractors have been demonstrated to be a powerful model compared with sequence labeling models. However, we discover that with the development of pre-trained language models, the performance of span extractors appears to become similar to sequence labeling models. To keep the advantages of span representation, we modified the model by under-sampling to balance the positive and negative samples and reduce the search space. Furthermore, we simplify the origin network architecture to combine the span extractor with BERT. Experiments demonstrate that even simplified architecture achieves the same performance and SEPT achieves a new state of the art result in scientific named entity recognition even without relation information involved.
BibTeX
@misc{mao2019sept,
  author = {Xian-Ling Mao},
  title = {{SEPT: Improving Scientific Named Entity Recognition with Span Representation}},
  year = {2019},
  eprint = {1911.03353},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1911.03353},
}
Figure