GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets

Wolfgang Otto, Matthäus Zloch, Lu Gan, Saurav Karmakar, Stefan DietzeEMNLP 2023 · DOI

Introduction

Scholarly Named Entity Recognition (NER) is a critical component for automating the analysis and synthesis of scientific literature. While general-purpose NER has reached high levels of maturity, extracting entities from scholarly texts remains a significant challenge. Existing datasets often treat "Machine Learning Models" as a monolithic category, frequently subsuming them under broad labels like "Methods." This lack of granularity prevents AI systems from distinguishing between specific model instances (e.g., "BERT-base"), general architectures (e.g., "Transformer"), and research methods (e.g., "Fine-tuning").

To address these limitations, researchers from GESIS and Heinrich-Heine-University have introduced GSAP-NER, a new gold-standard dataset and benchmark specifically designed for fine-grained extraction of machine learning models and datasets from scientific publications.

The GSAP-NER Annotation Schema

The GSAP-NER corpus employs a nuanced, 10-type tag set categorized into three primary groups:

  • MLModel-related: MLModel, ModelArchitecture, MLModelGeneric (informal), Method, and Task.
  • Dataset-related: Dataset, DatasetGeneric (informal), and DataSource.
  • Miscellaneous: ReferenceLink and URL.

A distinguishing feature of GSAP-NER is its focus on "informal mentions." In scientific writing, authors frequently use descriptors like "our model" or "the baseline" to refer to specific entities. By labeling these as generic types (e.g., MLModelGeneric), GSAP-NER provides the necessary data to support downstream tasks like co-reference resolution, linking informal descriptions back to their formal named counterparts.

Examples of entity annotations in GSAP-NER, illustrating the distinction between formal model names (e.g., XLM-R) and informal mentions (e.g., "Our model").

Methodology and Dataset Construction

Unlike many scholarly datasets that rely solely on abstracts, GSAP-NER is built on full-text annotations. The authors annotated 100 computer science publications from arXiv, resulting in a corpus of over 54,000 entity mentions. This full-text approach is vital for capturing the complex interplay between models, datasets, and results as they are developed throughout a paper's narrative.

The dataset also supports nested annotations. For example, a span like "the ResNets" can be labeled as an informal MLModelGeneric while the internal "ResNets" is simultaneously identified as a ModelArchitecture. This structural depth allows for a more accurate representation of linguistic nuances in technical writing.

Experimental Results and Baseline Performance

To establish a performance baseline, the authors fine-tuned several state-of-the-art transformer models, including SciBERT, RoBERTa (Base and Large), and SciDeBERTa-CS. Evaluation was conducted using 10-fold cross-validation, measuring both exact-match and partial-match F1 scores across all entity types.

Performance of baseline transformer models on the GSAP-NER dataset, with SciDeBERTa-CS emerging as the top performer.

SciDeBERTa-CS emerged as the strongest baseline, achieving an overall exact-match F1 of 0.64 and a partial-match F1 of 0.73. Analysis of individual entity performance revealed that concrete entities were significantly easier for models to identify than conceptual ones:

  • Dataset mentions achieved the highest performance (0.81 F1).
  • MLModel mentions followed with a 0.70 F1.
  • ModelArchitecture proved the most challenging (0.33 F1), likely due to its conceptual overlap with specific model names and general methods.

Key Insights and Discussion

The results highlight the linguistic ambiguity inherent in scholarly communication. While specific names are relatively easy to identify, the role of a technique—whether it is being used as a specific model, a general architecture, or a broader method—remains difficult for current AI systems to disambiguate.

The inclusion of informal mentions is a significant step forward. By providing the "connective tissue" between different parts of a paper, GSAP-NER enables the development of models that can maintain context across long documents. Additionally, learning curve experiments suggested that model performance begins to stabilize after approximately 40 fully annotated publications, offering a useful benchmark for future annotation efforts in specialized domains.

Conclusion and Future Work

GSAP-NER provides a robust foundation for more sophisticated scholarly AI. While currently focused on Computer Science, the methodology is extensible to other data-intensive fields. Future research will focus on leveraging external knowledge bases to improve entity disambiguation and exploring relationship extraction to map the connections between the identified models and datasets.

The GSAP-NER dataset and code are publicly available at https://data.gesis.org/gsap/gsap-ner.

Summary updated

BibTeX
@inproceedings{otto2023gsap,
  author = {Wolfgang Otto and Matthäus Zloch and Lu Gan and Saurav Karmakar and Stefan Dietze},
  title = {{GSAP-NER: A Novel Task, Corpus, and Baseline for Scholarly Entity Extraction Focused on Machine Learning Models and Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.548},
}
Figure