How Well Do Large Language Models Extract Keywords? A Systematic Evaluation on Scientific Corpora

Nacef Ben Mansour, Hamed Rahimi, Motasem AlrahabiNAACL 2025

TL;DR

This paper systematically evaluates how well Large Language Models (LLMs) extract keywords from scientific articles compared to traditional methods. They introduce a novel evaluation framework that combines exact and fuzzy matching to account for variations in keyword normalization. The findings show LLMs significantly outperform traditional approaches, especially when considering titles, highlighting their potential for revolutionizing scientific information retrieval.

What Problem Does This Paper Address?

Automatic keyword extraction is crucial for organizing scientific literature, improving search engines, and mapping research trends. However, existing methods, like statistical and graph-based approaches, often struggle with the complexities of scientific text, such as specialized jargon, cross-disciplinary ambiguity, and the dynamic nature of scientific language. While recent advancements have seen the rise of LLMs, their effectiveness and associated costs for keyword extraction, particularly on scientific corpora, haven't been thoroughly investigated. This paper aims to bridge this gap by providing a comprehensive empirical comparison of traditional and LLM-based keyword extraction methods.

Methods

The authors compare two main paradigms for keyword extraction:

  • Generative Approaches (LLMs): They utilize LLMs in a zero-shot learning framework. This means the LLMs are given a prompt to extract keywords without prior specific training on keyword extraction tasks. The prompt is designed to instruct the LLM to act as a keyword extraction master, focusing solely on identifying the most relevant keywords present in the text.
  • Embedding-Based Approaches: This approach leverages pre-trained language models (like BERT) to create dense vector representations (embeddings) of words and phrases. Keyword extraction is then performed by measuring the semantic similarity between the document's embedding and candidate keyword embeddings. The paper specifically uses KeyBERT, with two configurations: a default cosine similarity approach and an enhanced variant that incorporates Maximal Marginal Relevance (MMR) for diversity and Max Sum Distance (MSum) for novelty.
  • Traditional Approaches: These include well-established methods like TextRank, PositionRank, SingleRank, MultipartiteRank, TopicRank, and YAKE!. These methods rely on statistical measures, graph structures, or syntactic patterns to identify keywords.

To evaluate these methods, they employ two matching techniques:

  • Exact Matching: This is a strict evaluation where only identical terms between the extracted keywords and the author-provided keywords are considered matches. The F1-score (harmonic mean of precision and recall) is used as the primary metric.
  • Fuzzy Matching: This is a more flexible approach that allows for variations in keywords, such as plural forms or minor typos. They use the Levenshtein distance (edit distance) to quantify the similarity between extracted and reference keywords, allowing for a tolerance of up to 4 edits.

Datasets

The study uses a multilingual dataset constructed from the HAL database, an open archive for scientific publications in French and English. The dataset was compiled using the HAL API and initially contained around 12,000 articles. After filtering out duplicates and articles missing abstracts or keywords, the final corpus consists of 4,700 usable articles. The authors converted all titles, keywords, and abstracts to lowercase for consistent processing.

Key Insights and Ideas

  • LLMs are the new kings of keyword extraction: The paper strongly suggests that LLMs, even in a zero-shot setting, significantly outperform traditional methods in terms of precision and relevance.
  • Titles matter (for LLMs): Including article titles in the input significantly boosts LLM performance, often by around 10%, without a substantial increase in computational cost. This highlights the value of metadata.
  • Scale and architecture are key for LLMs: Larger LLMs (like LLaMA 3.1 70B) and certain architectures tend to perform better, but there's a trade-off with computational cost.
  • Prompting is critical for LLMs: Models like Gemma 7B showed poor performance due to strict penalties for not adhering to the exact output format specified in the prompt, underscoring the importance of prompt engineering.
  • Fuzzy matching is essential for a fair comparison: The Levenshtein distance-based fuzzy matching reveals that LLMs maintain their advantage even when accounting for linguistic variations, which is crucial for real-world applications.
  • Cost-efficiency is a growing concern: The paper introduces the Token Efficiency Score (TES) to balance performance with computational cost, identifying models like Claude 3 Haiku and LLaMA variants as cost-effective leaders.

Key Results

  • LLM-based methods consistently achieve superior precision and relevance compared to traditional approaches, as evidenced by higher F1-scores in both exact and fuzzy matching evaluations.
  • Including titles in the input generally improves F1-scores for LLMs by approximately 10%, while having a more modest impact on traditional and KeyBERT methods.
  • Top-performing LLMs include LLaMA 3.1 70B, Claude 3 Haiku, and LLaMA 3.1 8B, demonstrating the benefits of scale and advanced architectures.
  • Traditional methods show significant performance disparities, with gaps exceeding 60% between the best (PositionRank, MultipartiteRank) and worst (TextRank) performers.
  • KeyBERT demonstrates robustness across different input settings (abstract vs. abstract+title).

Limitations and Future Work

  • Limitations of LLMs:
    • Domain Specificity: LLMs' generic pre-training can limit precision on highly specialized scientific domains, necessitating costly fine-tuning.
  • Opacity: Their "black-box" nature makes interpretability and traceability challenging.
  • Stochastic Instability: Outputs can fluctuate based on prompts, requiring iterative prompt engineering.
  • Bias in Evaluation Corpora: Datasets like HAL might not fully represent all keyword extraction scenarios (e.g., keywords not present in abstracts/titles).
  • Future Work:
    • Prompt Engineering: Developing standardized templates and techniques to stabilize LLM outputs and reduce format variability.
  • Domain Adaptation: Fine-tuning LLMs on domain-specific corpora to improve precision.
  • Expanding to Full Text: Evaluating LLMs on full scientific articles, not just abstracts and titles.
  • Richer Evaluation: Incorporating metrics beyond F1-score, such as semantic coherence and thematic organization, and exploring broader contextual signals.
  • Scalability and Interpretability: Addressing the computational overhead and improving the explainability of LLM-based keyword extraction.

Summary updated

BibTeX
@inproceedings{mansour2025how,
  author = {Nacef Ben Mansour and Hamed Rahimi and Motasem Alrahabi},
  title = {{How Well Do Large Language Models Extract Keywords? A Systematic Evaluation on Scientific Corpora}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
}
Figure