TA-DA: Topic-Aware Domain Adaptation for Scientific Keyphrase Identification and Classification (Student Abstract)

Razvan-Alexandru Smadu, George-Eduard Zaharia, Andrei-Marius Avram, Dumitru-Clementin Cercel, Mihai Dascalu, Florin Pop

Introduction

Scientific Keyphrase Identification and Classification (SKIC) is a specialized form of Named Entity Recognition (NER) focused on extracting technical terms from research papers and categorizing them into types such as "Process," "Method," or "Material." The primary challenge in SKIC is the "data hunger" of deep learning models combined with the extreme specialization of scientific literature. High-quality annotated datasets are scarce, and models often struggle with "domain shifts"—for instance, a model trained on biology papers may fail when applied to computer science literature due to differing vocabularies and contexts.

The TA-DA Framework

To address these challenges, the authors propose the TA-DA (Topic-Aware Domain Adaptation) framework. This approach aims to make models robust to domain shifts without requiring massive amounts of new labeled data. The architecture is built on a SciBERT backbone followed by BiLSTM layers to capture sequence-level information.

The TA-DA framework overview, illustrating the integration of LDA-based topic discovery and adversarial perturbation.

The framework utilizes several key innovations:

  1. Multi-Task Learning: The model splits into two branches: one for identification (using a Linear-chain CRF) and one for classification (using a Softmax layer).
  2. Topic-Aware Domain Adaptation: Since scientific papers rarely have explicit domain labels, the authors use Latent Dirichlet Allocation (LDA) to automatically discover latent topics. A Gradient Reversal Layer (GRL) then forces the feature extractor to ignore topic-specific noise, focusing instead on the underlying structure of keyphrases.
  3. Adversarial Training: Using the Fast Gradient Sign Method (FGSM), the model adds small perturbations to the latent space during training. This "stress-tests" the model, improving its performance on real-world, noisy data.

Key Results and Discussion

The framework was evaluated on three major benchmarks: SemEval-2017 Task 10, SciERC (SciIE), and ACL RD-TEC 2.0. The researchers compared various latent topic representations, including LSA, NMF, and K-Means, finding that LDA consistently provided the most robust results.

Comparison of TA-DA variants against baselines on SemEval, SciIE, and ACL RD-TEC datasets.

As shown in the performance table, the DA-Adv variant achieved state-of-the-art or near-state-of-the-art scores across all datasets. On the ACL RD-TEC 2.0 dataset, it reached an F1-score of 82.5 for identification (KI) and 70.0 for classification (KIC), outperforming standard SciBERT and MTL-LSTM baselines.

Beyond raw performance, the model is remarkably efficient. The task-specific component contains only 2.4 million trainable parameters, significantly less than a full SciBERT fine-tune (109M parameters). The results also highlight a synergy between domain adaptation and adversarial training; while domain adaptation alone can sometimes cause slight performance dips, the inclusion of adversarial examples consistently pushes F1-scores higher by helping the model find stable, generalized features.

Conclusion

The TA-DA framework demonstrates that treating LDA-generated topics as domains is an effective strategy for generalizing across scientific fields. By combining this with adversarial training, the model achieves high accuracy and parameter efficiency. Future research will explore hierarchical representations to better capture the nested structure of scientific documents, such as sentences within specific sections.

Summary updated

Abstract

Keyphrase identification and classification is a Natural Language Processing and Information Retrieval task that involves extracting relevant groups of words from a given text related to the main topic. In this work, we focus on extracting keyphrases from scientific documents. We introduce TA-DA, a Topic-Aware Domain Adaptation framework for keyphrase extraction that integrates Multi-Task Learning with Adversarial Training and Domain Adaptation. Our approach improves performance over baseline models by up to 5% in the exact match of the F1-score.
BibTeX
@inproceedings{smadu2023ta,
  author = {Razvan-Alexandru Smadu and George-Eduard Zaharia and Andrei-Marius Avram and Dumitru-Clementin Cercel and Mihai Dascalu and Florin Pop},
  title = {{TA-DA: Topic-Aware Domain Adaptation for Scientific Keyphrase Identification and Classification (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27027},
}
Figure