Improving Absent Keyphrase Generation with Diversity Heads

Edwin Thomas, Sowmya VajjalaNAACL 2024 · DOI

TL;DR

This paper introduces "diversity heads" to improve the generation of absent keyphrases (those not explicitly in the text) by capturing more varied contextual representations. By treating present keyphrase extraction and absent keyphrase generation as distinct tasks, the authors achieve state-of-the-art results on multiple datasets, demonstrating a significant step forward in generating more relevant and diverse keyphrases.

What Problem Does This Paper Address?

Keyphrase Generation (KPG) is crucial for tasks like document indexing and information retrieval, aiming to automatically extract or generate representative phrases for a given text. A key distinction is made between "present" keyphrases (directly found in the text) and "absent" keyphrases (which are semantically relevant but not explicitly stated). While current models often train on both types together, this paper argues for separating these tasks. A recurring challenge in KPG, particularly for absent keyphrases, is generating diverse and non-repetitive phrases, often leading to a trade-off between precision and recall. This work aims to address this diversity issue and improve overall KPG performance by proposing a novel architecture.

Methods

The authors propose a two-pronged approach:

  1. Separating Present Keyphrase Extraction and Absent Keyphrase Generation:
    • Present Keyphrase Extraction: Treated as a standard sequence labeling task using the BIO encoding scheme. To handle class imbalance (many "other" tokens), they employ a weighted negative log-likelihood loss function.
  2. Absent Keyphrase Generation: Modeled as a sequence-to-set generation problem, adapting the transformer architecture. To achieve permutation invariance during training, they use the Hungarian algorithm for target realignment.
  3. Introducing "Diversity Heads":
    • The core novelty lies in a modified cross-attention layer within the absent keyphrase generation model. This layer incorporates "diversity heads," which are specialized attention modules.
  4. These heads learn different encoder representations for each decoder keyphrase unit, allowing the model to capture a wider range of contextual views for the same input. This aims to increase the diversity of generated keyphrases without relying on computationally expensive search techniques like beam search during inference.

Datasets

The paper uses a combination of datasets for training and evaluation:

  • Training Datasets:
    • KP20K: A large dataset of ~530k scientific paper abstracts.
  • KPTIMES: A dataset of ~290k news articles.
  • Testing Datasets:
    • KP20K (Test Split)
    • KPTIMES (Test Split)
    • INSPEC: 500 computer science abstracts.
  • SEMEVAL: 100 full-length articles.
  • KRAPIVIN: Full text computer science articles.
  • NUS: Full text scientific articles.

The authors note that KP20K primarily consists of abstracts, while KPTIMES includes full text, which leads to performance differences discussed later.

Key Insights and Ideas

  • Task Separation is Beneficial: Treating present keyphrase extraction and absent keyphrase generation as independent tasks, rather than jointly, leads to improved performance on both, especially for longer documents.
  • Diversity Heads for Absent Keyphrases: The proposed "diversity heads" are crucial for improving the generation of absent keyphrases, leading to better recall and diversity without complex decoding strategies.
  • Dataset Domain Matters: Performance varies significantly between abstract-based datasets (like KP20K) and full-text datasets (like KPTIMES), particularly for absent keyphrase generation. This suggests that the richness of the input text plays a vital role.
  • Annotation Quality Impacts Performance: Datasets with more structured and verified annotation processes (like KPTIMES) tend to yield better results across models, highlighting the importance of dataset quality.
  • Evaluation Metrics Nuance: The paper points out that F1@M (based on predicted keyphrase count) and F1@O (based on ground truth count) capture different aspects of performance, and their relative values can offer insights into model behavior.

Key Results

  • State-of-the-Art Performance: The proposed model achieves state-of-the-art results on five out of six datasets for absent keyphrase generation (in terms of F1@O) and on four out of six for present keyphrase extraction.
  • Absent Keyphrase Improvement: The model outperforms baselines by a significant margin, especially on datasets with full-text documents. For example, on NUS, the absent keyphrase performance for the model was 7.72 F1@M and 6.82 F1@O, outperforming others.
  • Diversity Heads Efficacy: Statistical significance tests confirm that the approach with diversity heads is significantly better than plain sequence-to-sequence or sequence-to-set generation models.
  • Parameter Scaling vs. Architecture: Scaling up baseline models by increasing parameters did not match the performance of the diversity heads approach, suggesting the architectural innovation is key.
  • Greedy Decoding Sufficiency: The diversity heads enable competitive performance with greedy decoding, avoiding the latency of beam search.

Limitations and Future Work

  • English-Centric and Supervised: The experiments are limited to English datasets and supervised learning scenarios.
  • Cross-Domain Performance: The model struggles with cross-domain generalization (e.g., training on KP20K and testing on KPTIMES).
  • Surface vs. Semantic Comparison: The evaluation measures primarily focus on surface-level matches rather than semantic similarity, potentially overlooking semantically closer but lexically different keyphrases.
  • Dataset Coverage Issues: Some datasets may have labeling inconsistencies or coverage issues where model-generated keyphrases are valid but not present in the ground truth.
  • Task Interdependence: While separated, extraction and generation tasks share commonalities that a multi-task framing might better exploit.
  • Future Directions: The authors suggest exploring cross-domain transfer, extending to other languages, and further investigating the nuances of evaluation measures and dataset characteristics. They also highlight that KPG performance, even with LLMs, remains lower than other NLP tasks, underscoring the need for continued research.

Summary updated

BibTeX
@inproceedings{thomas2024improving,
  author = {Edwin Thomas and Sowmya Vajjala},
  title = {{Improving Absent Keyphrase Generation with Diversity Heads}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.findings-naacl.102},
}
Figure