Lexicon-constrained Copying Network for Chinese Abstractive Summarization

Boyan Wan, Zhuo Tang, Li YangarXiv 2020 · arXiv · OpenReview

TL;DR

This paper introduces a novel "lexicon-constrained copying network" for Chinese abstractive summarization. It addresses the challenge of Chinese text's lack of explicit word delimiters by enabling the model to copy multi-character words directly, not just individual characters, leading to more efficient and accurate summaries.

What Problem Does This Paper Address?

Abstractive summarization aims to generate concise summaries that capture the essence of a source text, often by rephrasing and synthesizing information. Sequence-to-sequence models with a "copy mechanism" have been successful in English, allowing models to directly copy words from the source text into the summary. However, applying this to Chinese is problematic because Chinese sentences lack explicit word delimiters (like spaces). This forces most existing Chinese summarization models to operate at the character level. While this avoids segmentation errors, it makes copying meaningful multi-character words inefficient and often impossible, as models can only copy individual characters. This leads to less fluent and potentially less accurate summaries, especially when important multi-character words are present in the source.

Methods

The core innovation is a lexicon-constrained copying network (LCCN).

  • Multi-granularity Input: The encoder processes both individual characters and identified multi-character "potential words" from the source text. This is achieved by creating a "word-lattice" and feeding it into a Transformer encoder, allowing it to learn representations at both character and word levels.
  • Lexicon-Constrained Decoding: The decoder is designed to copy either a single character or a multi-character word at each step. Crucially, the copying of multi-character words is constrained by a lexicon of potential words derived from off-the-shelf Chinese word segmentators. This ensures that the model only attempts to copy valid multi-character words.
  • Word-Enhanced Beam Search: To handle the variable length of copied tokens (characters vs. words), a specialized beam search algorithm is proposed. This algorithm intelligently manages hypotheses containing both character and word tokens, improving efficiency and parallel computation during inference.
  • Word Selector (Optional): The model can optionally incorporate a "word selector" module, often leveraging a pre-trained language model like BERT. This module identifies keywords in the source text, which are then prioritized for copying by the decoder, further enhancing summary quality.

Datasets

  • LCSTS (Large Scale Chinese Social Media Text Summarization Dataset): This dataset, containing short Chinese social media texts and their summaries, is used for training and evaluation. The dataset is divided into parts, with scoring for relevance used to filter training data.

Key Insights and Ideas

  • The "Delimiter Problem" is Key: The lack of explicit word delimiters in Chinese is the fundamental hurdle for effective abstractive summarization using copy mechanisms.
  • Multi-granularity is Essential: To overcome this, models need to handle information at multiple levels – characters and meaningful word units.
  • Constrained Copying is Powerful: Simply allowing character-level copying is insufficient. Constraining the copying process to recognized multi-character words significantly improves performance.
  • Search Algorithm Matters: The standard beam search needs adaptation to handle variable-length token copying, and the proposed word-enhanced beam search is a practical solution.
  • External Knowledge Boosts Performance: Integrating keyword information, especially from powerful pre-trained models like BERT, provides a significant performance boost.

Key Results

  • The proposed LCCN model outperforms previous character-based abstractive summarization models.
  • When combined with the word selector, the LCCN achieves state-of-the-art performance on the LCSTS dataset for ROUGE-1 and ROUGE-2 scores, and is competitive for ROUGE-L.
  • The word-enhanced beam search component provides a significant performance boost (e.g., +1.7 ROUGE-1, +1.0 ROUGE-2).
  • Qualitative examples show that the LCCN can generate more faithful and accurate summaries, correctly copying multi-character words and capturing nuances like the motive for a crime.

Limitations and Future Work

  • Word Segmentation Dependency: The model relies on off-the-shelf word segmentators. Errors or limitations in these segmentators could impact the lexicon used for copying.
  • Repetition: While the model aims to reduce repetition, the LCCN+word selector variant showed a slightly higher repetition rate, possibly due to a smaller output space.
  • Future Work: The authors suggest applying their model to other tasks like comment generation and to other languages. They also plan to explore further integration of external knowledge.

Summary updated

Abstract

Copy mechanism allows sequence-to-sequence models to choose words from the input and put them directly into the output, which is finding increasing use in abstractive summarization. However, since there is no explicit delimiter in Chinese sentences, most existing models for Chinese abstractive summarization can only perform character copy, resulting in inefficient. To solve this problem, we propose a lexicon-constrained copying network that models multi-granularity in both encoder and decoder. On the source side, words and characters are aggregated into the same input memory using a Transformerbased encoder. On the target side, the decoder can copy either a character or a multi-character word at each time step, and the decoding process is guided by a word-enhanced search algorithm that facilitates the parallel computation and encourages the model to copy more words. Moreover, we adopt a word selector to integrate keyword information. Experiments results on a Chinese social media dataset show that our model can work standalone or with the word selector. Both forms can outperform previous character-based models and achieve competitive performances.
BibTeX
@misc{wan2020lexicon,
  author = {Boyan Wan and Zhuo Tang and Li Yang},
  title = {{Lexicon-constrained Copying Network for Chinese Abstractive Summarization}},
  year = {2020},
  eprint = {2010.08197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.08197},
}
Figure