TL;DR
This paper introduces FEW-TK, the first dataset specifically designed for few-shot scientific typed keyphrase recognition, addressing the scarcity of labeled data in this domain. They propose a novel, portable annotation schema and demonstrate the challenge of this task by evaluating several state-of-the-art models, highlighting the need for further research.
What Problem Does This Paper Address?
Extracting key information from scientific literature is crucial for various downstream tasks like relation extraction and question answering. Named Entity Recognition (NER) is a fundamental technique for this, but traditional NER definitions are too restrictive for scientific terms. While "keyphrase extraction" is often used, it typically focuses only on identifying terms, not classifying them into meaningful types. Furthermore, scientific NER is data-intensive, requiring large amounts of labeled data, which is scarce. This paper tackles this by introducing "Typed Keyphrases" (TK) – a broader concept encompassing scientific terms with assigned types – and focusing on the "few-shot" setting. Few-shot learning allows models to generalize to new, unseen classes with only a few examples, making it ideal for low-resource scenarios like scientific IE. The authors aim to create a dataset and evaluation framework for this specific challenge.
Methods
The core contribution is the creation of the FEW-TK dataset and a novel annotation schema. The schema defines 9 coarse-grained keyphrase types (e.g., Focus, Proposed Technique, Algorithm/Tool, Study Domain, Supplementary Material, Dataset, Metric, Allied Terms, Performance) and 38 fine-grained types. This hierarchical structure allows for portability to other scientific domains by using only the coarse-grained types. The dataset consists of 500 AI/ML abstracts, annotated at the abstract level to capture a distilled representation of the paper.
For evaluation, the authors employ standard few-shot learning methodologies. They adapt the INTRA and INTER settings from FEW-NERD. In the INTRA setting, different coarse-grained entity types are used for training, validation, and testing to simulate learning entirely new categories. In the INTER setting, fine-grained categories are shared across splits. They evaluate several state-of-the-art few-shot NER models, including ProtoBERT, NNShot, StructShot, CONTAINER, and MAML-ProtoNet, using both BERT and SciBERT as backbones.
Datasets
The primary dataset introduced is FEW-TK, comprising 500 AI/ML abstracts from four sub-domains: Natural Language Processing (NLP), classical Artificial Intelligence (AI) and Machine Learning (ML), Data Mining and Information Retrieval, and Computer Vision (CV). Abstracts were sourced from reputable journals.
The paper also references and compares performance against existing scientific NER datasets like SCIERC (Luan et al., 2018) and FEW-NERD (Ding et al., 2021) for context and benchmarking.
Key Insights and Ideas
- Typed Keyphrases (TK) as a more suitable concept: The authors argue that "Typed Keyphrases" are a more appropriate term than "Named Entities" for scientific literature, as they encompass a broader range of important scientific terms that may not be proper nouns.
- Portability of the annotation schema: The hierarchical nature of the proposed schema, with coarse-grained types, makes it adaptable to other scientific domains beyond AI/ML.
- FEW-TK is a challenging benchmark: Even in the fully supervised setting, models perform significantly worse on FEW-TK compared to other scientific NER datasets, indicating the difficulty introduced by the expanded keyphrase set.
- Few-shot performance is significantly lower: State-of-the-art few-shot models struggle considerably on FEW-TK, highlighting the need for specialized models and techniques for this task.
- Unexpected backbone performance: In some few-shot scenarios, the general BERT model outperformed the domain-specific SciBERT, suggesting that domain adaptation for few-shot learning is complex.
Key Results
- Fully Supervised Setting: SciBERT-Tagger achieved an F1 score of 48.91 on FEW-TK, which is notably lower than its performance on SCIERC (65.81), underscoring the dataset's difficulty.
- Few-Shot Setting (INTRA): CONTAINER (SciBERT) achieved the highest F1 score of 39.11 in the 3-way 1-shot setting.
- Few-Shot Setting (INTER): CONTAINER (SciBERT) also performed best in the 5-way 1-shot setting with an F1 score of 50.35.
- MAML-ProtoNet showed very low performance: This was attributed to issues in its span detection component.
- Span Error Analysis: The NNShot model showed higher false negative (FN) errors than false positive (FP) errors, indicating a tendency to miss keyphrases rather than incorrectly identify them.
- Type Error Analysis: Within errors (misclassifying fine-grained types while correctly identifying coarse-grained ones) were more prevalent than outer errors (misclassifying coarse-grained types).
Limitations and Future Work
- Discontinuous Spans: The current annotation scheme does not handle discontinuous spans as a single keyphrase. For example, "rule-based and neural models" would be annotated as two separate keyphrases ("rule-based" and "neural models") rather than a single one. The authors aim to address this in future work.
- Need for further research: The low performance of existing models on FEW-TK clearly indicates that significant work is needed to develop effective models for few-shot scientific typed keyphrase recognition.
- Domain-specific challenges: The paper notes that while abstracts from allied AI fields often contain entities from pure AI/ML, the reverse is not always true, suggesting potential biases in data distribution across domains.
- Descriptive nature of scientific terms: The descriptive manner in which scientific terms are often presented makes precise demarcation of keyphrases challenging.
Summary updated