Keyphrase Generation with Cross-Document Attention

Shizhe Diao, Yan Song, Tong Zhang

TL;DR

This paper introduces CDKGen, a novel keyphrase generation model that leverages cross-document attention to incorporate information from related documents, significantly improving keyphrase quality. By combining this with a Transformer architecture and a copy mechanism, CDKGen achieves state-of-the-art performance across multiple benchmark datasets, generating more accurate and diverse keyphrases.

What Problem Does This Paper Address?

Keyphrase generation aims to automatically extract concise phrases that summarize the core content of a document. Traditional approaches often use encoder-decoder models, typically based on Transformers, that focus solely on the input document. This limitation means they miss out on crucial "corpus-level" information – the latent topics and dependencies that exist across a collection of related documents. This can lead to keyphrases that are less informative, especially for "absent" keyphrases (those not explicitly present in the text) or when dealing with out-of-vocabulary (OOV) words. The paper addresses this by proposing a method to incorporate knowledge from relevant external documents to enhance keyphrase generation.

Methods

The core of CDKGen is a Transformer-based encoder-decoder architecture. The novelty lies in two key components:

  • Cross-Document Attention: This mechanism allows the model to attend to relevant documents from a larger corpus. It learns to represent latent topics from these documents and uses this information to guide the decoder. This is achieved by creating "key" and "value" vectors for each document in the collection, and then using the input document's representation as a "query" to find relevant topics.
  • Copy Mechanism: To handle out-of-vocabulary (OOV) words that might be crucial for keyphrases, CDKGen incorporates a pointer-generator network. This allows the model to directly copy words from the input document, supplementing its ability to generate words from a predefined vocabulary.

These components are integrated into the Transformer architecture, allowing the model to leverage both the internal structure of the input document and the external knowledge from related documents.

Datasets

The authors evaluated CDKGen on five benchmark datasets commonly used in keyphrase generation research:

  • INSPEC: Contains journal paper abstracts with keyphrases assigned by professional indexers.
  • NUS: A scientific dataset with full papers and student-annotated keyphrases.
  • KRAPIVIN: Includes full papers from ACM with author-provided and reviewer-verified keyphrases.
  • SEMEVAL: A dataset of full papers with keyphrases from the ACM Digital Library.
  • KP20K: A large dataset of paper abstracts from various online resources.

The paper also mentions that for consistency with prior work, they pre-processed KP20K to remove duplicates that appear in other datasets.

Key Insights and Ideas

  • The Power of Context: The most significant insight is that looking beyond a single document is crucial for effective keyphrase generation. Relevant external documents provide valuable topic information that can significantly improve both present and absent keyphrase prediction.
  • Synergistic Effect: The combination of cross-document attention and the copy mechanism is more than the sum of its parts. They effectively complement each other, with cross-document attention providing broader topic context and the copy mechanism handling specific word generation challenges.
  • Transformer's Adaptability: The Transformer architecture proves to be a robust backbone, readily adaptable to incorporate cross-document attention and copy mechanisms for improved performance.
  • Efficiency Matters: CDKGen achieves state-of-the-art results without resorting to complex training procedures like reinforcement learning, making it a more efficient solution.

Key Results

CDKGen achieved state-of-the-art performance across all five benchmark datasets, demonstrating significant improvements over existing models.

  • Overall Performance: CDKGen consistently outperformed baselines like TRANSFORMER, TRANS+COPY, and TRANS+CD. For instance, cross-document attention alone (TRANS+CD) improved performance by an average of 3.70% F₁@O over TRANSFORMER.
  • Present Keyphrases: The copy mechanism proved particularly effective for present keyphrases, helping to generate words that might otherwise be OOV.
  • Absent Keyphrases: CDKGen showed substantial gains in predicting absent keyphrases, with notable improvements on datasets like KRAPIVIN, where external documents provided more relevant candidate words.
  • Generation Diversity: The cross-document attention mechanism was shown to increase the diversity of generated keyphrases, as visualized by t-SNE plots of decoding states. CDKGen produced the most unique predictions among the models.
  • Number of Keyphrases: CDKGen achieved the lowest Mean Absolute Error (MAE) in predicting the correct number of keyphrases, indicating better control over output length.

Limitations and Future Work

  • Annotation Quality: The paper notes that the performance gains from cross-document attention can be influenced by the annotation quality of the datasets. Datasets with more professional annotators (like INSPEC and KRAPIVIN) showed larger gains, suggesting that annotators might implicitly use external document information.
  • Copy Mechanism Effectiveness: The copy mechanism's effectiveness is tied to the presence of relevant words in the input document. For datasets with fewer present keyphrases or where OOV words are less critical for absent keyphrases, its impact might be less pronounced.
  • Future Work: While not explicitly detailed, the success of CDKGen suggests that further exploration of integrating external knowledge sources and refining attention mechanisms for cross-document understanding could lead to even more advanced keyphrase generation systems. The authors also highlight the potential for further research into generating more diverse keyphrases.

Summary updated

Abstract

Keyphrase generation aims to produce a set of phrases summarizing the essentials of a given document. Conventional methods normally apply an encoder-decoder architecture to generate the output keyphrases for an input document, where they are designed to focus on each current document so they inevitably omit crucial corpus-level information carried by other similar documents, i.e., the cross-document dependency and latent topics. In this paper, we propose CDKGen, a Transformer-based keyphrase generator, which expands the Transformer to global attention with cross-document attention networks to incorporate available documents as references so as to generate better keyphrases with the guidance of topic information. On top of the proposed Transformer + cross-document attention architecture, we also adopt a copy mechanism to enhance our model via selecting appropriate words from documents to deal with out-of-vocabulary words in keyphrases. Experiment results on five benchmark datasets illustrate the validity and effectiveness of our model, which achieves the state-of-the-art performance on all datasets. Further analyses confirm that the proposed model is able to generate keyphrases consistent with references while keeping sufficient diversity. The code of CDKGen is available at https://github.com/SVAIGBA/CDKGen.
BibTeX
@misc{diao2020keyphrase,
  author = {Shizhe Diao and Yan Song and Tong Zhang},
  title = {{Keyphrase Generation with Cross-Document Attention}},
  year = {2020},
  eprint = {2004.09800},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2004.09800},
}
Figure