Keyphrase Extraction with Incomplete Annotated Training Data

Yanfei Lei, Chunming Hu, Guanghui Ma, Richong Zhang

Introduction

Keyphrase Extraction (KPE) is a fundamental task in natural language processing aimed at identifying the most salient phrases within a document. While state-of-the-art models typically approach KPE as a supervised classification task, they often suffer from the "incomplete annotation" problem. In real-world datasets, human annotators frequently miss valid keyphrases. Under the standard "closed-world assumption," these unlabeled but valid phrases are treated as negative examples, leading to biased models that are penalized for correct predictions.

Proposed Methodology

To address these challenges, this research introduces a robust neural architecture combined with a novel training strategy based on negative sampling.

Neural Architecture

The model utilizes a RoBERTa sequence encoder to generate contextualized word embeddings. These embeddings are processed by a sliding CNN with multiple kernel sizes (1 to 5) to capture features for various n-grams. Finally, a binary classifier determines the likelihood of each candidate n-gram being a keyphrase.

The proposed KPE architecture featuring a RoBERTa encoder and a multi-scale sliding CNN for n-gram feature extraction.

Negative Sampling Strategy

The core innovation lies in the training process. Instead of treating all unlabeled phrases as negative instances, the authors employ negative sampling. By randomly selecting a small subset of unlabeled phrases as negatives in each training step, the model reduces the probability of repeatedly penalizing itself for identifying valid but unlabeled keyphrases. This approach also mitigates the extreme class imbalance inherent in KPE, where non-keyphrases vastly outnumber true keyphrases.

Experimental Results

The model was evaluated across scientific (Kp20k, Inspec, etc.) and open-domain (OpenKP) datasets.

Robustness to Incomplete Labels

To simulate varying degrees of annotation quality, the researchers created synthetic datasets by "masking" (removing) valid labels at rates ranging from 10% to 90%. As shown in the performance analysis, the negative sampling approach maintains significantly higher F1 scores compared to standard training as the masking rate increases.

F1@5 score comparison between standard training (no sampling) and negative sampling across various masking rates, demonstrating superior robustness to missing labels.

Performance Highlights

  • Open Domain: On the OpenKP dataset, the model achieved an F1@1 of 0.380, surpassing the previous state-of-the-art RoBERTa-Chunk model (0.355).
  • Scientific Domain: The method remains competitive on well-annotated scientific benchmarks, indicating that negative sampling does not degrade performance even when labels are relatively complete.

Conclusion

By relaxing the closed-world assumption through negative sampling, this work provides a more resilient framework for keyphrase extraction in noisy, real-world environments. Future research will focus on modeling the dependencies between overlapping candidate phrases and implementing intelligent gating mechanisms for negative sample selection.

Summary updated

BibTeX
@inproceedings{lei2021keyphrase,
  author = {Yanfei Lei and Chunming Hu and Guanghui Ma and Richong Zhang},
  title = {{Keyphrase Extraction with Incomplete Annotated Training Data}},
  booktitle = {W-NUT},
  year = {2021},
  doi = {10.18653/v1/2021.wnut-1.4},
}
Figure