Abstract
Automatic keyphrase extraction (AKE) is a critical task for document summarization and information retrieval. This paper re-examines AKE by addressing two primary bottlenecks: rigid candidate selection and the lack of systematic feature evaluation. By introducing linguistically-informed candidate selection rules and evaluating 25 features in both supervised and unsupervised settings, the authors demonstrate that unsupervised methods can achieve performance levels comparable to expensive, human-trained supervised models.
Introduction
Keyphrase extraction involves identifying the most salient terms in a document. Despite its importance, traditional systems often suffer from poor coverage due to restrictive candidate selection patterns. Furthermore, the proliferation of proposed features without comparative analysis makes it difficult to discern which are truly effective. This study addresses these gaps by refining how candidates are identified and re-evaluating feature utility across different learning paradigms.
Methodology
Enhanced Candidate Selection
The authors argue that standard noun-phrase (NP) patterns are insufficient for capturing the variety of ways concepts are expressed in scientific text. They propose six specific rules to improve candidate coverage, accounting for semantic equivalents like possessives and prepositional phrases.
Key innovations include:
- Alternation Rules: Recognizing that "agent's goal" is semantically equivalent to "goal of agent."
- Prepositional Phrases: Specifically targeting "of-PP" forms (e.g., "quality of service") and allowing them to be longer (up to 4 words) than standard NPs.
- Frequency Heuristics: Applying different thresholds for single words (frequency $\ge 2$) versus multi-word phrases (frequency $\ge 1$).
Feature Engineering Re-evaluation
The study evaluates 25 features, categorized into document cohesion (e.g., TF*IDF, position), keyphrase cohesion (e.g., section co-occurrence), and term cohesion (internal phrase strength). To enable unsupervised extraction, the authors utilized Google n-gram counts to calculate IDF, leveraging web-scale data to determine word rarity based on the entire web rather than a small local corpus.
Datasets and Evaluation
The research utilized 250 scientific articles from the ACM Digital Library. A significant finding was that author-provided keyphrases often do not appear verbatim in the text or are only present as substrings. To create a more reliable ground truth, computer science students were hired to manually annotate the papers, resulting in a "reader-assigned" keyphrase set used for benchmarking.
Key Insights
- Linguistic Flexibility: Incorporating alternations (possessives and "of" phrases) significantly improves candidate coverage. A system cannot extract a phrase it does not first identify as a candidate.
- Unsupervised Stability: Features traditionally used in supervised learning, such as section information and phrase length, remain highly effective in unsupervised settings.
- Substrings and TF: Counting substrings (e.g., counting "grid computing" whenever "grid computing algorithm" appears) provides a more accurate measure of term frequency (TF) than exact matching alone.
Key Results
The evaluation compared a supervised Maximum Entropy classifier against an unsupervised model. The results indicate that while supervised methods lead in performance, the gap is narrow enough to justify unsupervised approaches when labeled data is unavailable.
As shown in the results, the supervised model achieved a match count of 3.15 at the top 15 candidates, compared to 2.61 for the unsupervised version. The most consistently powerful features were section information and candidate length, while the impact of TF*IDF was relatively minor when structural features (like title overlap and section location) were present.
Limitations and Future Work
The study's focus on Computer Science articles may limit the generalizability of certain features, such as acronyms or specific POS sequences, to other domains like news. Future research could investigate more sophisticated weighting schemes for unsupervised feature combination and explore the inclusion of more complex linguistic structures, such as those containing adverbs or verbs, which were intentionally ignored in this study to maintain simplicity.
Summary updated