Introduction
Identifying and classifying keyphrases in scientific literature—such as labeling "BERT" as a Model—is a cornerstone of scientific text processing. This task, known as Scholarly Keyphrase Boundary Classification (SKBC), is vital for applications like automated summarization and advanced literature search. However, SKBC is particularly challenging due to the highly specialized nature of scientific language and the scarcity of expert-annotated data. While Multi-Task Learning (MTL) offers a way to leverage auxiliary linguistic tasks, it often suffers from "catastrophic forgetting," where the model loses general linguistic features while specializing in a specific target task.
Methodology
To address these challenges, we introduce a novel framework that integrates Multi-Task Learning with Knowledge Distillation (KD) and a specific structural constraint designed to preserve representational integrity.
The framework consists of three main components:
- Teacher Models: Eleven "single-task" teacher models are trained on various auxiliary NLP tasks, including Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and natural language inference from the GLUE benchmark.
- Student Model: A multi-task model based on SciBERT is trained to perform the target SKBC task while simultaneously distilling knowledge from all eleven teachers.
- Cosine Embedding Constraint: Beyond standard output-level distillation, we introduce a "cosine embedding constraint." This forces the student model to mimic the teachers' internal hidden layer representations by minimizing the cosine distance between their embeddings. This ensures the student retains rich, specialized linguistic knowledge across tasks and prevents divergence from the teachers' specialized understanding.
Experimental Setup
The framework was evaluated on three benchmark scholarly datasets:
- SemEval 2017 Task 10: Spanning Computer Science, Physics, and Material Science.
- ACL RD-TEC 2.0: Focused on the Computational Linguistics domain.
- SciIE: Targeting entity recognition in Artificial Intelligence.
Results and Discussion
The proposed model (SciBERT + MTL + KD + Cosine) achieved state-of-the-art results across all datasets. On SemEval 2017, it reached an F1-score of 75.08 for identification and 57.08 for classification, significantly outperforming the vanilla SciBERT baseline (66.70 / 48.21).
Representational Alignment and Confidence
A key insight is that representational alignment—imitating how a teacher "sees" a word—is as crucial as imitating the final label. This acts as a powerful regularizer. Furthermore, entropy analysis reveals that the proposed model is not only more accurate but also more certain in its predictions.
As shown in Figure 5, the proposed model consistently exhibits lower entropy across various entity types compared to the baseline, indicating higher certainty in its classifications. This suggests that the embedding constraints help the model form more robust decision boundaries.
Conclusion
This work demonstrates that combining multi-task learning with embedding-constrained knowledge distillation effectively mitigates catastrophic forgetting in specialized domains. While the model shows remarkable performance, challenges remain in handling long, complex keyphrases and the inherent subjectivity of human labeling. Future research will explore the framework's efficacy in few-shot and semi-supervised learning scenarios.
Summary updated