Animesh Prasad, Min-Yen Kan. Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017). 2017.
Abstract
This paper presents the WING-NUS team's approach to the SemEval-2017 ScienceIE task, which involves extracting and categorizing keyphrases from scientific literature. By framing the problem as a joint sequence labeling task using Conditional Random Fields (CRFs), the authors demonstrate that identifying a phrase and its scientific role simultaneously is more effective than treating them as isolated steps.
The Challenge of Scientific Keyphrase Extraction
Extracting information from scientific papers is notoriously difficult compared to general web text. Scientific prose is dense with jargon and complex structures, presenting three specific challenges:
- High Density: Keyphrases appear much more frequently in scientific excerpts.
- Overlapping/Nested Phrases: Terms are often nested (e.g., "blocks" inside "equally sized blocks").
- Context-Dependency: The same phrase can change roles (Material, Process, or Task) depending on the sentence context.
Traditional "top-K" extraction methods often fail because they do not account for the sequential flow of text or the relationships between neighboring words.
Methodology: Joint Sequence Labeling with CRFs
The researchers employed Conditional Random Fields (CRFs) to model identification and classification as a joint task. Instead of first finding a phrase and then labeling it, the CRF labels each token in a sequence with both its boundary and its type (e.g., B-Material, I-Material).
Feature Engineering and the Power of Context
The model uses a robust set of surface-level features, including N-gram prefixes/suffixes, Part-of-Speech (POS) tags, and orthographic features (capitalization, digits). Crucially, the model incorporates the "O" Feature—the predicted label of the previous token.
Feature ablation studies (Figure 1) highlight that the previous token output is the most critical feature. When this contextual signal is removed, performance drops significantly, proving that scientific keyphrases are highly dependent on their immediate linguistic neighbors.
Key Results and Insights
The study's primary finding is that Joint Modeling Wins. The authors compared their joint model against "Individual" expert models (one model per category). As shown in Figure 3, the Joint model performed significantly better across all metrics, suggesting that the signal of a phrase's type helps the model accurately identify phrase boundaries.
- Subtask A (Identification): Achieved an $F_1$ score of 0.46.
- Subtask B (Classification): Achieved an $F_1$ score of 0.33.
- End-to-End Pipeline: The top configuration reached an $F_1$ of 0.27 for combined extraction and relation identification.
Limitations and Future Work
Despite its success, the model faced several hurdles. Standard CRFs struggle with nested phrases because they typically assign only one label per token, leading to lower recall. Additionally, the model struggled with the "Task" category (0.17 $F_1$), likely due to a distribution shift between training and test data and the inherent complexity of task-related phrases.
Future work will likely involve using deep learning or semantic embeddings to better handle nested entities and improve relation extraction, which currently lacks deep semantic understanding.
Summary updated