A Comparison of Supervised Keyphrase Extraction Models

Florin Adrian Bulgarov, Cornelia CarageaWWW 2015 · DOI

TL;DR

This paper compares supervised keyphrase extraction models, finding that incorporating citation context significantly boosts performance. They introduce "CeKE," a model that leverages both document content and citation networks, and demonstrate its superiority over existing methods, particularly when augmented with a "keyphraseness" feature.

What Problem Does This Paper Address?

The explosion of digital information necessitates efficient ways to summarize documents. Keyphrases serve as concise topic descriptors, crucial for applications like document classification, recommendation, and search. While both supervised and unsupervised methods exist for keyphrase extraction, a comprehensive comparison of the leading supervised approaches, especially those leveraging richer contextual information beyond just the document's text, has been lacking. This paper aims to fill that gap by comparing a novel citation-context-aware model with established state-of-the-art supervised methods.

Methods

The paper focuses on supervised keyphrase extraction, where candidate phrases are classified as either keyphrases or non-keyphrases. The core of their contribution lies in the CeKE (Citation Enhanced Keyphrase Extraction) model. CeKE extends traditional supervised models by incorporating features derived from citation networks. This means it doesn't just look at the text of a paper but also at how it's cited by other papers and how it cites others.

Key features used in CeKE include:

  • Document-based features: Term frequency (TF), TF-IDF, relative position of the phrase, and part-of-speech tags.
  • Citation network features: TF-IDF calculated within citation contexts and boolean features indicating if a phrase appears in cited or citing contexts.

They also explore an enhanced version of CeKE by adding the keyphraseness feature, which measures how often a candidate phrase appears as a tag or keyphrase in the training data. For classification, they use a Naïve Bayes classifier.

Datasets

The experiments were conducted on datasets comprising research papers (titles, abstracts, and citation contexts) from two top-tier machine learning conferences:

  • World Wide Web (WWW)
  • Knowledge Discovery and Data Mining (KDD)

The author-annotated keyphrases from the PDFs of these papers were used as the gold standard for evaluation.

Key Insights and Ideas

  • Citation context is king: The most significant insight is that leveraging citation networks dramatically improves keyphrase extraction performance. This highlights that a paper's role within the broader academic discourse provides valuable clues about its core topics.
  • Beyond local neighborhoods: Traditional methods often rely solely on a document's text or its immediate textual neighbors. This paper demonstrates the power of looking at a document's position within a larger, interconnected citation graph.
  • Keyphraseness adds value: The "keyphraseness" feature, which captures how frequently a phrase is recognized as a keyphrase in training data, further boosts the performance of the citation-context-aware model.
  • A clear winner emerges: When comparing various supervised models, the citation-context-based approaches, particularly CeKE with the keyphraseness feature, consistently outperform others.

Key Results

The paper presents a quantitative comparison of different models using precision, recall, and F1-score. The results show:

  • CeKE + keyphraseness achieved the highest F1-scores for both the WWW (0.322) and KDD (0.321) datasets.
  • This enhanced CeKE model significantly outperformed established methods like Maui (0.285/0.280), Hulth (0.129/0.172), and KEA (0.168/0.145).
  • While Maui achieved relatively high recall, its precision was the lowest among the tested methods.
  • CeKE alone (without keyphraseness) also showed strong performance (0.285/0.280), demonstrating the inherent benefit of citation context.
  • Attempts to incorporate external Wikipedia-based features did not yield notable improvements and sometimes decreased performance.

Limitations and Future Work

  • Limited feature exploration: The authors mention experimenting with Wikipedia-based features but note that performance did not improve. It's possible other external knowledge sources or more sophisticated feature engineering from Wikipedia could be beneficial.
  • Classifier choice: While Naïve Bayes was used for consistency, exploring other classifiers might yield further improvements.
  • Dataset scope: The study was limited to two specific machine learning conference datasets. Evaluating on a broader range of domains and document types would be valuable.
  • Future work: The paper implicitly suggests further exploration of citation-context features and potentially combining them with other advanced NLP techniques for even better keyphrase extraction.

Summary updated

BibTeX
@inproceedings{bulgarov2015comparison,
  author = {Florin Adrian Bulgarov and Cornelia Caragea},
  title = {{A Comparison of Supervised Keyphrase Extraction Models}},
  booktitle = {The Web Conference},
  year = {2015},
  doi = {10.1145/2740908.2742776},
}
Figure