Supervised Keyphrase Extraction as Positive Unlabeled Learning

Lucas Sterckx, Cornelia Caragea, Thomas Demeester, Chris Develder

Abstract

Supervised keyphrase extraction typically treats non-selected phrases as negative examples, leading to "noisy negatives" due to human subjectivity. We reframe this task as Positive Unlabeled (PU) Learning. By employing a two-stage reweighting strategy with normalization and coreference features, our approach treats unlabeled phrases as potentially positive. Experimental results across five diverse datasets demonstrate significant improvements in extraction performance, nearly matching models trained on consensus labels from multiple annotators.

Introduction: The Subjectivity Problem

Keyphrase extraction—the automatic identification of the most important phrases in a document—is traditionally modeled as a binary classification task. However, this setup suffers from a fundamental flaw: subjectivity. When a single annotator (e.g., a paper's author) labels keyphrases, they select only a small subset of valid candidates. In a standard supervised framework, every other candidate phrase is treated as a "negative" example.

This creates "noisy negatives"—phrases that are semantically appropriate but were simply overlooked or ignored by a specific individual. As shown in Figure 2, the vast majority of keyphrases are selected by only a small fraction of annotators, confirming that "ground truth" is highly fluid.

Distribution of user agreement on keyphrases, highlighting the high degree of subjectivity.

Methodology: Positive Unlabeled Learning

To address the noise in negative labels, we propose a Positive Unlabeled (PU) Learning framework. Instead of assuming non-labeled phrases are negative, we treat them as "Unlabeled," acknowledging they could be positive or negative.

We implement a two-stage reweighting strategy:

  1. Initial Classification: A classifier is trained on the noisy, single-annotator data to establish baseline probabilities.
  2. Reweighting: The initial model estimates the likelihood that an unlabeled phrase is actually a keyphrase. Unlabeled examples are then duplicated in the training set: one copy is labeled positive with weight $w$, and the other negative with weight $1-w$.

We further refine this process with document-level normalization and coreference features. Normalization accounts for varying document lengths, while coreference resolution identifies unlabeled phrases semantically related to known keyphrases (e.g., linking "Louis Michel" to "Prime Minister Louis Michel"), assigning them higher positive weights.

Experimental Results

The proposed method was evaluated on five datasets: Online News, Lifestyle Magazines, WWW, KDD, and Inspec. These range from crowdsourced news articles to academic abstracts.

As detailed in Figure 5, our "Reweighting + Norm + Coref" method consistently outperforms standard supervised baselines and basic PU learning implementations.

Performance comparison of the proposed PU learning method against baselines across five datasets.

On the Online News dataset, the Macro Average $F_1$ (MAF1) score improved from .364 to .374, approaching the .381 performance achieved by models trained on multiple human annotators. Precision@5 also saw consistent gains, particularly in the academic domains (WWW and KDD), demonstrating the robustness of the PU approach in handling unbalanced data where keyphrases are sparse relative to candidates.

Conclusion

Reframing keyphrase extraction as a PU learning problem effectively mitigates the impact of subjective labeling. By treating unlabeled candidates as potentially positive through a principled reweighting scheme, we can train more robust models that better capture the consensus of multiple human perspectives, even when only single-annotator data is available. Future work may extend this framework to other subjective NLP tasks like extractive summarization.

Summary updated

BibTeX
@inproceedings{sterckx2016supervised,
  author = {Lucas Sterckx and Cornelia Caragea and Thomas Demeester and Chris Develder},
  title = {{Supervised Keyphrase Extraction as Positive Unlabeled Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1198},
}
Figure