Improving Performance of Automatic Keyword Extraction (AKE) Methods Using PoS-Tagging and Enhanced Semantic-Awareness

Enes Altuncu, Jason R. C. Nurse, Xu Yang, Jie Guo, Shujun LiarXiv 2022 · DOI · arXiv

Abstract

Automatic keyword extraction (AKE) is a critical component of modern information retrieval and natural language processing, facilitating tasks such as text summarization, topic analysis, and document indexing. Despite its importance, many existing AKE methods struggle to balance statistical frequency with linguistic and semantic relevance. This paper presents a universal, post-processing-based approach designed to enhance the performance of any AKE method. By integrating Part-of-Speech (PoS) tagging, domain-specific thesauri, and Wikipedia-based named entity recognition, the proposed framework re-evaluates candidate keywords to ensure they align with human-centric selection patterns. Experimental results across 17 datasets and five state-of-the-art (SOTA) AKE methods demonstrate significant and consistent performance gains, with an average F1-score improvement of 25.8%.

Introduction

As digital textual data continues to grow exponentially, the ability to automatically extract meaningful keywords has become indispensable. However, AKE remains a challenging task because there is no universal consensus on what constitutes a "keyword." Current SOTA methods often fall into two categories: purely statistical models that ignore linguistic structure, and models that lack semantic awareness, failing to recognize technical concepts or established entities.

This research addresses these gaps by introducing a "plug-and-play" post-processor. This framework does not require retraining existing models; instead, it refines their output by applying linguistic filters and external knowledge bases to better mimic human keyword selection.

Methodology

The proposed approach consists of a three-step post-processing pipeline that re-evaluates the scores of candidate keywords generated by a baseline AKE method:

1. PoS-Tagging Filter

Analysis of human-selected keywords reveals that over 90% follow specific linguistic patterns, primarily consisting of nouns and noun phrases. The post-processor utilizes PoS-tagging to identify the grammatical structure of candidates. If a candidate does not fit these established linguistic profiles (e.g., it is a lone verb or adverb), its score is "zeroed out," effectively filtering out noise that statistical methods often misidentify as significant.

2. Context-Aware Thesauri

To incorporate domain-specific knowledge, the system checks if a candidate keyword appears in specialized thesauri, such as MeSH (Medical Subject Headings) or AGROVOC (Agriculture). When a match is found, the candidate's importance score is doubled. To make this process scalable, the researchers implemented an Automatic Context Identification step, using a machine-learning classifier to automatically select the most relevant thesaurus for a given document.

3. Wikipedia Named Entities

Wikipedia serves as a massive, context-independent repository of human-defined concepts. The framework treats Wikipedia page titles as a proxy for "keyword-worthiness." Candidates matching Wikipedia entries (excluding common stop-words) receive a score boost, ensuring that recognized entities are prioritized in the final extraction list.

Experimental Setup

The framework was evaluated using 17 diverse English-language datasets, including scientific abstracts (Inspec, PubMed), news (KPCrowd), and miscellaneous benchmarks (SemEval 2010 & 2017). To test the universality of the approach, it was applied to five distinct SOTA baseline algorithms:

  • YAKE!
  • KP-Miner
  • RaKUn
  • LexRank
  • SIFRank+

Key Insights and Discussion

The Wikipedia Overlap

A significant finding of the study was that 64.39% of human-selected "gold standard" keywords are Wikipedia page titles. This high correlation justifies the use of Wikipedia as a powerful, general-purpose filter for identifying high-quality keywords across various domains.

Linguistic Constraints

The research confirmed that human keyword selection is highly constrained by length and grammar. Over 82% of human-selected keywords are unigrams or bigrams, and 94.7% are three words or fewer. By enforcing these length constraints and PoS rules, the post-processor successfully eliminated "junk" phrases often picked up by language-agnostic statistical models like YAKE!.

Universal Compatibility

Because the method only requires a list of candidates and their initial scores, it can be "bolted on" to any AKE method. This makes it a highly flexible tool for researchers and practitioners who wish to improve existing pipelines without the computational cost of retraining models.

Results

The experimental results were consistently positive across all tested configurations:

  • Significant F1 Gains: The combined approach improved the F1-score of baseline methods by an average of 25.8%.
  • Algorithm-Specific Success: The most dramatic improvement was observed with RaKUn, which saw performance increases of up to 53.8%.
  • Consistency: In some configurations, the method improved performance in 100% of the cases tested, demonstrating its robustness across different datasets and document types.
  • Automation Viability: The automatic context classifier performed nearly as well as manual thesaurus selection, proving the system's readiness for unsupervised, large-scale applications.

Conclusion and Future Work

This paper demonstrates that a simple, semantic-aware post-processing framework can significantly enhance the accuracy of Automatic Keyword Extraction. While currently focused on English, the logic of the approach is language-independent and could be adapted to other languages by updating PoS patterns and thesauri. Future research will explore more sophisticated Word Sense Disambiguation (WSD) to handle polysemy, ensuring that score boosts are applied only when the context of a word matches its definition in external knowledge bases.

Summary updated

Abstract

Automatic keyword extraction (AKE) has gained more importance with the increasing amount of digital textual data that modern computing systems process. It has various applications in information retrieval (IR) and natural language processing (NLP), including text summarisation, topic analysis and document indexing. This paper proposes a simple but effective post-processing-based universal approach to improve the performance of any AKE methods, via an enhanced level of semantic-awareness supported by PoS-tagging. To demonstrate the performance of the proposed approach, we considered word types retrieved from a PoS-tagging step and two representative sources of semantic information - specialised terms defined in one or more context-dependent thesauri, and named entities in Wikipedia. The above three steps can be simply added to the end of any AKE methods as part of a post-processor, which simply re-evaluate all candidate keywords following some context-specific and semantic-aware criteria. For five state-of-the-art (SOTA) AKE methods, our experimental results with 17 selected datasets showed that the proposed approach improved their performances both consistently (up to 100% in terms of improved cases) and significantly (between 10.2% and 53.8%, with an average of 25.8%, in terms of F1-score and across all five methods), especially when all the three enhancement steps are used. Our results have profound implications considering the ease to apply our proposed approach to any AKE methods and to further extend it.
BibTeX
@misc{altuncu2022improving,
  author = {Enes Altuncu and Jason R. C. Nurse and Xu Yang and Jie Guo and Shujun Li},
  title = {{Improving Performance of Automatic Keyword Extraction (AKE) Methods Using PoS-Tagging and Enhanced Semantic-Awareness}},
  year = {2022},
  eprint = {2211.05031},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.05031},
}
Figure