Learning Syntactic and Dynamic Selective Encoding for Document Summarization

Haiyang Xu, Yahao He, Kun Han, Junwen Chen, Xiangang Li

Abstract

Text summarization aims to generate a headline or a short summary consisting of the major information of the source text. Recent studies employ the sequence-to-sequence framework to encode the input with a neural network and generate abstractive summary. However, most studies feed the encoder with the semantic word embedding but ignore the syntactic information of the text. Further, although previous studies proposed the selective gate to control the information flow from the encoder to the decoder, it is static during the decoding and cannot differentiate the information based on the decoder states. In this paper, we propose a novel neural architecture for document summarization. Our approach has the following contributions: first, we incorporate syntactic information such as constituency parsing trees into the encoding sequence to learn both the semantic and syntactic information from the document, resulting in more accurate summary; second, we propose a dynamic gate network to select the salient information based on the context of the decoder state, which is essential to document summarization. The proposed model has been evaluated on CNN/Daily Mail summarization datasets and the experimental results show that the proposed approach outperforms baseline approaches.

Introduction

Abstractive text summarization relies heavily on Sequence-to-Sequence (Seq2Seq) neural networks to compress long documents into short, readable summaries. However, standard Seq2Seq approaches suffer from two major blind spots:

  1. Ignoring Syntax: Most models treat the source text as a flat sequence of semantic word embeddings, ignoring the underlying grammatical structure (like noun phrases or attributive clauses). This often leads to summaries that misattribute actions or mangle original meanings.
  2. Static Information Filtering: While previous models introduced "selective gates" to filter out noise, these gates were static, deciding importance once during encoding. In reality, what is "salient" changes at every step of generating a summary.

This paper improves abstractive document summarization by injecting structural syntax into the encoding process and introducing a "dynamic selective gate" that adapts to the decoder's current state.

Proposed Methodology

The authors propose a novel architecture that tackles structural and salience issues by modifying the encoder and the gating mechanism:

Syntactic Sequence Encoder

Instead of feeding raw words into a BiLSTM, the text is first parsed into a constituency tree and serialized using a depth-first traversal. The input sequence becomes a mix of actual words and syntactic parsing symbols (e.g., NP for Noun Phrase, VP for Verb Phrase). The hidden states of the parsing symbols are max-pooled to create a global "syntactic vector" that guides the decoder.

Dynamic Selective Gates

To control the flow of information, the authors introduce a parameterized gate network. Unlike previous static gates, this gate is dynamic—it takes the current decoder state as an input. This allows the model to open or close the gate to specific source words on the fly, depending on what part of the summary is currently being written.

Model architecture featuring the syntactic sequence encoder and dynamic selective gate.

The model builds on the standard pointer-generator network, calculating attention using the new syntactic vector and the dynamically gated hidden states.

Experimental Setup

The model was evaluated on the CNN/Daily Mail dataset. Crucially, the authors used the original version of the dataset (retaining actual entity names) rather than the anonymized version. This makes the task more realistic and challenging, as it requires the model to handle out-of-vocabulary words effectively.

Key Insights and Analysis

  • Syntax prevents semantic blending: By embedding parse tree symbols directly into the sequence, words belonging to different syntactic subtrees are separated. This weakens artificial connections between adjacent but grammatically unrelated words, preventing grammatically confused summaries.
  • Salience is a moving target: The dynamic gate proves that adjusting the filter at every decoding step is highly effective. Visualizations show it successfully suppresses stop words and boosts key entities exactly when the decoder needs to output them.
  • Synergy between syntax and dynamic gating: Ablation studies reveal that while adding just the syntactic encoder or just the dynamic gate improves performance, combining them yields the best results. Syntax provides the structural map, and the dynamic gate navigates it step-by-step.

Results

The proposed model outperformed several strong baselines, including standard Seq2Seq with attention, Graph-based models, and Pointer-generator networks with static Selective Gates.

Performance comparison on the CNN/Daily Mail dataset.

On the combined CNN/Daily Mail test set, the model achieved a ROUGE-1 score of 40.37, compared to 39.53 for the standard Pointer-generator + coverage baseline. The model's performance remained highly stable across varying document lengths, even when tested on documents between 1000 and 1400 words.

Conclusion and Future Work

The authors acknowledge that serializing a parse tree flattens the data into a 1D sequence, which may not capture the full hierarchical structure. Additionally, the dynamic selective gate currently ignores parsing symbols during the gating process.

Future work will explore Graph Neural Networks or Tree-LSTMs to model true hierarchical structures and integrate parsing symbols into the dynamic gating mechanism for even finer control over information flow.

Summary updated

Abstract

Text summarization aims to generate a headline or a short summary consisting of the major information of the source text. Recent studies employ the sequence-to-sequence framework to encode the input with a neural network and generate abstractive summary. However, most studies feed the encoder with the semantic word embedding but ignore the syntactic information of the text. Further, although previous studies proposed the selective gate to control the information flow from the encoder to the decoder, it is static during the decoding and cannot differentiate the information based on the decoder states. In this paper, we propose a novel neural architecture for document summarization. Our approach has the following contributions: first, we incorporate syntactic information such as constituency parsing trees into the encoding sequence to learn both the semantic and syntactic information from the document, resulting in more accurate summary; second, we propose a dynamic gate network to select the salient information based on the context of the decoder state, which is essential to document summarization. The proposed model has been evaluated on CNN/Daily Mail summarization datasets and the experimental results show that the proposed approach outperforms baseline approaches.
BibTeX
@inproceedings{xu2019learning,
  author = {Haiyang Xu and Yahao He and Kun Han and Junwen Chen and Xiangang Li},
  title = {{Learning Syntactic and Dynamic Selective Encoding for Document Summarization}},
  booktitle = {IJCNN},
  year = {2019},
  url = {https://openreview.net/forum?id=qeiu2S4BHKA},
}
Figure