Bidirectional Attentional Encoder-Decoder Model and Bidirectional Beam Search for Abstractive Summarization

Kamal Al‐Sabahi, Zuping Zhang, Kang YangarXiv 2018 · DOI · arXiv

Abstract

Sequence generative models with RNN variants, such as LSTM and GRU, show promising performance on abstractive document summarization. However, they still have some issues that limit their performance, especially while dealing with long sequences. One of the issues is that all current models employ a unidirectional decoder, which reasons only about the past and is limited in retaining future context while giving a prediction. This makes these models suffer by generating unbalanced outputs. Moreover, unidirectional attention-based document summarization can only capture partial aspects of attentional regularities. To this end, we propose an end-to-end trainable bidirectional RNN model to tackle the aforementioned issues. The model has a bidirectional encoder-decoder architecture in which the encoder and the decoder are bidirectional LSTMs. In addition, a bidirectional beam search mechanism is proposed as an approximate inference algorithm for generating the output summaries. Experimental results on the CNN / Daily Mail dataset show that the proposed model outperforms current abstractive state-of-the-art models by a considerable margin.

Introduction

Most current abstractive summarization models rely on a unidirectional decoder. This approach is akin to writing a sentence where one can only look at the words already written to decide what comes next. If an error occurs early on—such as misidentifying a subject—that error cascades through the rest of the summary.

Furthermore, unidirectional models struggle with "future context." Consider the phrase "They said that Teddy...". Without future context, a model might struggle to predict the next word. However, if it also sees "...was a great President," the word "Roosevelt" becomes obvious. Standard RNN-based decoders lack this foresight, often leading to "unbalanced" outputs where the beginning of a summary is strong, but the end loses focus or misses essential information from the tail end of the source text.

Methodology

The authors propose an end-to-end trainable architecture with two primary innovations: a Bidirectional Encoder-Decoder and Bidirectional Beam Search (BBS).

Bidirectional Encoder-Decoder

While bidirectional encoders are standard, this model introduces a bidirectional LSTM decoder. It consists of a forward decoder (left-to-right) and a backward decoder (right-to-left). These are "cross-initialized": the forward decoder starts with the final state of the backward encoder, and vice versa. This ensures that even at the very first step of generation, the model has "seen" the entire input from both directions.

Figure 1: The proposed bidirectional attentional encoder-decoder architecture.

Bidirectional Beam Search (BBS)

Standard beam search explores the most likely next tokens moving forward. BBS is a two-stage inference algorithm:

  1. Backward Pass: It runs a backward beam search to generate a full "future" sequence.
  2. Forward Pass: It runs a forward beam search that calculates the probability of the next token by "jointly" considering the history (from the forward pass) and the predicted future (from the backward pass).

To handle real-world text challenges, the model integrates a pointer-generator network to copy rare words from the source and a coverage mechanism to prevent repetitive outputs. Additionally, the paper reaffirms that reversing the input sequence significantly improves LSTM performance by establishing a better communication link between the start of the input and the start of the output.

Experimental Results

The model was evaluated on the CNN / Daily Mail dataset, a standard benchmark for long-form document summarization. The training set consists of approximately 286,000 pairs, with source documents averaging 781 words and summaries averaging 56 words.

The proposed model, specifically the variant using reversed input and coverage (Bidir_Rev_Cov), achieved significant gains over the baselines.

Figure 3: Performance comparison of the proposed model against state-of-the-art baselines on the CNN/Daily Mail dataset.

As shown in Figure 3, the Bidir_Rev_Cov model achieved a ROUGE-1 score of 42.6%, outperforming the standard Pointer-Generator+Coverage baseline (39.5%). Similar improvements were noted in ROUGE-2 (18.8% vs 17.3%) and ROUGE-L (38.5% vs 36.4%). Qualitatively, the model proved much better at capturing the full context of a story, correctly identifying relationships between individuals that unidirectional models often misattributed.

Conclusion and Future Work

While the bidirectional decoder roughly doubles the number of parameters compared to a simple unidirectional one (34M vs 21M), the authors argue their BBS algorithm is more efficient than previous bidirectional search attempts, with a complexity of 3|B||V| compared to |B|^2|V|.

Future research will focus on exploring ways to encourage even higher levels of abstraction and novelty in the generated summaries, as current models still lean heavily on copying from the source. Additionally, there is a recognized need for new evaluation metrics that better capture the nuances of long-form summarization beyond simple n-gram overlap.

Summary updated

Abstract

Sequence generative models with RNN variants, such as LSTM, GRU, show promising performance on abstractive document summarization. However, they still have some issues that limit their performance, especially while deal-ing with long sequences. One of the issues is that, to the best of our knowledge, all current models employ a unidirectional decoder, which reasons only about the past and still limited to retain future context while giving a prediction. This makes these models suffer on their own by generating unbalanced outputs. Moreover, unidirec-tional attention-based document summarization can only capture partial aspects of attentional regularities due to the inherited challenges in document summarization. To this end, we propose an end-to-end trainable bidirectional RNN model to tackle the aforementioned issues. The model has a bidirectional encoder-decoder architecture; in which the encoder and the decoder are bidirectional LSTMs. The forward decoder is initialized with the last hidden state of the backward encoder while the backward decoder is initialized with the last hidden state of the for-ward encoder. In addition, a bidirectional beam search mechanism is proposed as an approximate inference algo-rithm for generating the output summaries from the bidi-rectional model. This enables the model to reason about the past and future and to generate balanced outputs as a result. Experimental results on CNN / Daily Mail dataset show that the proposed model outperforms the current abstractive state-of-the-art models by a considerable mar-gin.
BibTeX
@misc{alsabahi2018bidirectional,
  author = {Kamal Al‐Sabahi and Zuping Zhang and Kang Yang},
  title = {{Bidirectional Attentional Encoder-Decoder Model and Bidirectional Beam Search for Abstractive Summarization}},
  year = {2018},
  eprint = {1809.06662},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.06662},
}
Figure