Adaptive Beam Search Decoding for Discrete Keyphrase Generation

Xiaoli Huang, Tongge Xu, Lvan Jiao, Yueran Zu, Youmin ZhangAAAI 2021 · OpenReview

Abstract

Keyphrase Generation compresses a document into some highly-summative phrases, which is an important task in natural language processing. Most state-of-the-art adopt greedy search or beam search decoding methods. These two decoding methods generate a large number of duplicated keyphrases and are time-consuming. Moreover, beam search only predicts a fixed number of keyphrases for different documents. In this paper, we propose an adaptive generation model-AdaGM, which is mainly inspired by the importance of the first words in keyphrase generation. In AdaGM, a novel reset state training mechanism is proposed to maximize the difference in the predicted first words. To ensure the discreteness and get an appropriate number of keyphrases according to the content of the document adaptively, we equip beam search with a highly effective filter mechanism. Experiments on five public datasets demonstrate the proposed model can generate marginally less duplicated and more accurate keyphrases.

Introduction

Keyphrase generation is the task of condensing a long document into a few salient phrases, such as "machine learning" or "neural networks." This task is essential for information retrieval and document summarization. Current state-of-the-art models typically follow a "one-to-many" approach, generating a long sequence of phrases separated by delimiters. However, this approach faces three major hurdles: redundancy, where models repeat the same keyphrase; rigidity, where standard beam search produces a fixed number of phrases regardless of document content; and inefficiency, requiring large beam sizes that are computationally expensive.

Methodology

To address these challenges, the authors introduce the Adaptive Generation Model (AdaGM). AdaGM incorporates two primary innovations: a reset state training mechanism and an adaptive beam search with filtering.

Reset State Training Mechanism

Observations indicate that in approximately 90% of cases, different keyphrases for the same document start with different words. To capitalize on this, AdaGM modifies the training process. When the model finishes one phrase and transitions to the next, it "resets" the decoder's hidden state. This forces the model to utilize its attention mechanism to identify a new relevant part of the document rather than obsessing over the previously generated phrase.

Architecture of AdaGM with Reset State Training Mechanism

Adaptive Beam Search with Filtering

Traditional beam search is often rigid, producing a fixed number of candidates. AdaGM introduces an adaptive filtering mechanism. Instead of keeping a fixed number of candidates, the model evaluates the probability of the first word of every potential keyphrase. It only retains words that exceed a specific probability threshold ($\alpha$). This allows the model to adaptively determine the number of keyphrases a document requires based on the strength of the textual evidence.

Adaptive Beam Search with Filter Mechanism

The authors also explored two decoding strategies: AdaGM#, which is fully parallel and requires every phrase to start with a different word, and AdaGM, a semi-parallel version that allows for overlap if the model exhibits high confidence.

Experimental Results

The model was trained on the KP20k dataset, consisting of over 500,000 scientific abstracts, and evaluated across five standard benchmarks: KP20k, Inspec, Krapivin, NUS, and SemEval.

Performance and Accuracy

AdaGM outperformed previous state-of-the-art models, such as CatSeq and Kp-RL, across nearly all datasets. Notably, on the NUS dataset, it improved the F1@5 score by 17.8% over the previous best model.

Redundancy and Speed

The "DupRatio" (duplication ratio) dropped significantly. On the KP20k dataset, AdaGM's duplication rate was roughly 70% lower than standard competitive models. Furthermore, AdaGM is highly efficient, processing documents in an average of 89ms, compared to 320ms for the popular CopyRNN model—a 3.6x speedup.

Conclusion

AdaGM represents a significant advancement in keyphrase generation by prioritizing the diversity of starting words and implementing an adaptive filtering mechanism. By resetting the decoder state at phrase boundaries, the model treats each generation as a fresh sub-task, reducing noise and improving accuracy. Future work may involve incorporating structural or syntactic information, such as part-of-speech tags, to further refine the model's understanding of grammatical boundaries.

Summary updated

Abstract

Keyphrase Generation compresses a document into some highly-summative phrases, which is an important task in natural language processing. Most state-of-the-art adopt greedy search or beam search decoding methods. These two decoding methods generate a large number of duplicated keyphrases and are time-consuming. Moreover, beam search only predicts a fixed number of keyphrases for different documents. In this paper, we propose an adaptive generation model-AdaGM, which is mainly inspired by the importance of the first words in keyphrase generation. In AdaGM, a novel reset state training mechanism is proposed to maximize the difference in the predicted first words. To ensure the discreteness and get an appropriate number of keyphrases according to the content of the document adaptively, we equip beam search with a highly effective filter mechanism. Experiments on five public datasets demonstrate the proposed model can generate marginally less duplicated and more accurate keyphrases. The codes of AdaGM are available at: https://github.com/huangxiaolist/adaGM.
BibTeX
@inproceedings{huang2021adaptive,
  author = {Xiaoli Huang and Tongge Xu and Lvan Jiao and Yueran Zu and Youmin Zhang},
  title = {{Adaptive Beam Search Decoding for Discrete Keyphrase Generation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=JoqgTPz1kFK},
}

Loading related content...

Figure