Re-examining Automatic Keyphrase Extraction Approaches in Scientific Articles

Su Nam Kim, Min-Yen KanMWE@IJCNLP 2009 · OpenReview

Abstract

Automatic keyphrase extraction (AKE) is a critical task for document summarization and information retrieval. This paper re-examines AKE by addressing two primary bottlenecks: rigid candidate selection and the lack of systematic feature evaluation. By introducing linguistically-informed candidate selection rules and evaluating 25 features in both supervised and unsupervised settings, the authors demonstrate that unsupervised methods can achieve performance levels comparable to expensive, human-trained supervised models.

Introduction

Keyphrase extraction involves identifying the most salient terms in a document. Despite its importance, traditional systems often suffer from poor coverage due to restrictive candidate selection patterns. Furthermore, the proliferation of proposed features without comparative analysis makes it difficult to discern which are truly effective. This study addresses these gaps by refining how candidates are identified and re-evaluating feature utility across different learning paradigms.

Methodology

Enhanced Candidate Selection

The authors argue that standard noun-phrase (NP) patterns are insufficient for capturing the variety of ways concepts are expressed in scientific text. They propose six specific rules to improve candidate coverage, accounting for semantic equivalents like possessives and prepositional phrases.

Linguistic rules for candidate selection including frequency, length, and alternation heuristics.

Key innovations include:

  • Alternation Rules: Recognizing that "agent's goal" is semantically equivalent to "goal of agent."
  • Prepositional Phrases: Specifically targeting "of-PP" forms (e.g., "quality of service") and allowing them to be longer (up to 4 words) than standard NPs.
  • Frequency Heuristics: Applying different thresholds for single words (frequency $\ge 2$) versus multi-word phrases (frequency $\ge 1$).

Feature Engineering Re-evaluation

The study evaluates 25 features, categorized into document cohesion (e.g., TF*IDF, position), keyphrase cohesion (e.g., section co-occurrence), and term cohesion (internal phrase strength). To enable unsupervised extraction, the authors utilized Google n-gram counts to calculate IDF, leveraging web-scale data to determine word rarity based on the entire web rather than a small local corpus.

Datasets and Evaluation

The research utilized 250 scientific articles from the ACM Digital Library. A significant finding was that author-provided keyphrases often do not appear verbatim in the text or are only present as substrings. To create a more reliable ground truth, computer science students were hired to manually annotate the papers, resulting in a "reader-assigned" keyphrase set used for benchmarking.

Key Insights

  • Linguistic Flexibility: Incorporating alternations (possessives and "of" phrases) significantly improves candidate coverage. A system cannot extract a phrase it does not first identify as a candidate.
  • Unsupervised Stability: Features traditionally used in supervised learning, such as section information and phrase length, remain highly effective in unsupervised settings.
  • Substrings and TF: Counting substrings (e.g., counting "grid computing" whenever "grid computing algorithm" appears) provides a more accurate measure of term frequency (TF) than exact matching alone.

Key Results

The evaluation compared a supervised Maximum Entropy classifier against an unsupervised model. The results indicate that while supervised methods lead in performance, the gap is narrow enough to justify unsupervised approaches when labeled data is unavailable.

Performance comparison of the best feature sets in supervised (S) and unsupervised (U) configurations across different candidate counts.

As shown in the results, the supervised model achieved a match count of 3.15 at the top 15 candidates, compared to 2.61 for the unsupervised version. The most consistently powerful features were section information and candidate length, while the impact of TF*IDF was relatively minor when structural features (like title overlap and section location) were present.

Limitations and Future Work

The study's focus on Computer Science articles may limit the generalizability of certain features, such as acronyms or specific POS sequences, to other domains like news. Future research could investigate more sophisticated weighting schemes for unsupervised feature combination and explore the inclusion of more complex linguistic structures, such as those containing adverbs or verbs, which were intentionally ignored in this study to maintain simplicity.

Summary updated

BibTeX
@inproceedings{kim2009re,
  author = {Su Nam Kim and Min-Yen Kan},
  title = {{Re-examining Automatic Keyphrase Extraction Approaches in Scientific Articles}},
  booktitle = {MWE@IJCNLP},
  year = {2009},
  url = {https://openreview.net/forum?id=ieW0FzEsYmq},
}
Figure