SEKE: Specialised Experts for Keyword Extraction

Matej Martinc, Thi Hong Hanh Tran, Senja Pollak, Boshko KoloskiEMNLP 2025

TL;DR

This paper introduces SEKE, a novel keyword extraction method that leverages a Mixture of Experts (MoE) architecture built on top of the DeBERTa language model. SEKE achieves state-of-the-art results by allowing specialized "expert" networks to handle different aspects of text, improving keyword extraction performance, especially in scenarios with limited training data.

What Problem Does This Paper Address?

Keyword extraction is crucial for tasks like summarizing and indexing large amounts of text. While powerful language models (LLMs) have advanced NLP, they often struggle to match the performance of specialized supervised models for keyword extraction, especially when training data is scarce. Existing methods often treat all tokens uniformly, missing out on the potential for specialized processing of different linguistic features. This paper aims to bridge this gap by exploring how a Mixture of Experts (MoE) approach, which allows different parts of a model to specialize, can enhance supervised keyword extraction.

Methods

The core of SEKE is a novel token classification head that combines a Mixture of Experts (MoE) with a Bidirectional Long Short-Term Memory (BiLSTM) network, built upon the DeBERTa language model.

  • DeBERTa Backbone: A powerful pre-trained transformer model known for its strong performance on various NLP tasks.
  • Mixture of Experts (MoE): This technique uses a "gating network" to route tokens to specialized "expert" networks. Each expert can learn to focus on different patterns or features within the input. This allows the model to be more efficient and potentially more effective by not forcing a single network to handle everything.
  • BiLSTM Layer: Added to amplify sequential information that might be lost due to the MoE's token-level routing, especially important for multi-word keywords.
  • Noisy Top-k Gating: A specific MoE routing mechanism that introduces noise to prevent expert collapse and ensure load balancing, while selecting the top 'k' experts for processing.

The authors also explored different training scenarios, including fine-tuning on validation sets and full training on all available data, to understand the impact of data size.

Datasets

The experiments were conducted on six publicly available datasets:

  • KP20k: A large dataset of scientific articles.
  • Inspec: Scientific articles.
  • Krapivin: Scientific articles.
  • KPTimes: News articles.
  • JPTimes: News articles.
  • openKP: Web pages.

These datasets cover three domains: computer science, news, and web pages, and exhibit variability in the number of keywords per document.

Key Insights and Ideas

  • Specialization is Key: The study strongly suggests that allowing different parts of a neural network to specialize (via MoE) is highly beneficial for keyword extraction, leading to improved performance.
  • Domain-Specific Specialization: Experts tend to specialize differently based on the domain. In scientific texts, experts focus more on syntactic patterns, while in news and web data, they lean towards semantic clues like named entities.
  • MoE + BiLSTM Synergy: Combining MoE with BiLSTM layers shows a synergistic effect, leading to better performance than either component alone, especially in mitigating the loss of sequential information.
  • Data Efficiency: The MoE approach shows promise even with smaller datasets, and the addition of BiLSTM can further lower data requirements.

Key Results

  • State-of-the-Art Performance: SEKE (specifically the DeBERTa MoE BiLSTM configuration) achieved state-of-the-art results on several datasets, outperforming strong supervised and unsupervised baselines.
  • Improved Performance: Adding MoE layers consistently improved performance across most datasets, especially when combined with BiLSTM.
  • Expert Behavior: Analysis revealed that experts specialize in distinct linguistic features, such as punctuation, parts-of-speech, and named entities, depending on the dataset.
  • Long Keyword Extraction: The MoE component alone can struggle with longer keywords due to context slicing, but the addition of BiLSTM significantly mitigates this issue.

Limitations and Future Work

  • Language and Domain Generalization: The current evaluation is limited to English-language datasets from three domains. Future work should explore performance on other languages and more diverse domains.
  • Hyperparameter Tuning: The study used default hyperparameter values. More extensive tuning could potentially yield further performance gains.
  • Potential for Misuse: The authors acknowledge that effective keyword extraction tools could be exploited for malicious purposes like SEO spam.
  • Computational Cost: While MoE offers efficiency, the addition of BiLSTM increases model complexity and inference time, which could be a factor for very large-scale deployments.
  • Future Research: The authors plan to investigate the exact data thresholds for SEKE's effectiveness and adapt LLMs to specific corpora and keyword extraction regimes using fine-tuning and in-context learning. They also aim to explore new domains and languages.

Summary updated

BibTeX
@inproceedings{martinc2025seke,
  author = {Matej Martinc and Thi Hong Hanh Tran and Senja Pollak and Boshko Koloski},
  title = {{SEKE: Specialised Experts for Keyword Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure