TL;DR
This paper improves abstractive document summarization by injecting structural syntax into the encoding process and introducing a "dynamic selective gate." By serializing constituency parse trees and dynamically filtering source information based on the decoder's current state, the model better captures salient details and grammatical structure, achieving state-of-the-art results on the CNN/Daily Mail dataset.
What Problem Does This Paper Address?
Abstractive text summarization relies heavily on Sequence-to-Sequence (Seq2Seq) neural networks to compress long documents into short, readable summaries. However, standard Seq2Seq approaches suffer from two major blind spots:
- Ignoring Syntax: Most models treat the source text as a flat sequence of semantic word embeddings. They ignore the underlying grammatical structure (like noun phrases or attributive clauses), which often leads to summaries that misattribute actions or mangle the original meaning.
- Static Information Filtering: Documents are long and full of irrelevant details. While previous models introduced "selective gates" to filter out noise before passing data to the decoder, these gates were static. They decided what was important once during the encoding phase. But in reality, what is "salient" changes at every step of generating a summary.
Methods
The authors propose a novel architecture that tackles both problems by modifying the encoder and the gating mechanism:
- Syntactic Sequence Encoder: Instead of just feeding raw words into a BiLSTM, the authors first parse the text into a constituency tree. They serialize this tree using a depth-first traversal. The input sequence becomes a mix of actual words and syntactic parsing symbols (e.g., NP for Noun Phrase, VP for Verb Phrase). The hidden states of the parsing symbols are max-pooled to create a global "syntactic vector" that guides the decoder.
- Dynamic Selective Gates: To control the flow of information, the authors introduce a parameterized gate network. Unlike previous static gates, this gate is dynamic—it takes the current decoder state as an input. This allows the model to open or close the gate to specific source words on the fly, depending on what part of the summary is currently being written.
- Pointer-Generator with Syntactic Attention: The model builds on the standard pointer-generator network (which can copy words directly from the source to handle out-of-vocabulary terms). It calculates attention using the new syntactic vector and the dynamically gated hidden states.
Datasets
- CNN/Daily Mail: The model was trained and evaluated on the widely used CNN/Daily Mail dataset. Crucially, the authors used the original version of the dataset (which retains actual entity names) rather than the anonymized version (which replaces entities with unique identifiers). This makes the task more challenging and realistic, as it requires the model to handle out-of-vocabulary words effectively.
Key Insights and Ideas
- Syntax prevents semantic blending: By embedding parse tree symbols directly into the sequence, words that belong to different syntactic subtrees are separated by more parsing symbols. This weakens the artificial connection between adjacent but grammatically unrelated words, preventing the model from generating grammatically confused summaries.
- Salience is a moving target: The dynamic gate proves that adjusting the filter at every decoding step is highly effective. Visualizations of the gate show that it successfully suppresses stop words ("the", "is") and boosts key entities exactly at the time step when the decoder needs to output them.
- Synergy between syntax and dynamic gating: Ablation studies reveal that while adding just the syntactic encoder or just the dynamic gate improves performance over baselines, combining them yields the best results. The syntax provides the structural map, and the dynamic gate navigates it step-by-step.
Key Results
The proposed model outperformed several strong baselines, including standard Seq2Seq with attention, Graph-based models, and Pointer-generator networks with static Selective Gates.
- On the combined CNN/Daily Mail test set, the model achieved:
- ROUGE-1: 40.37 (vs. 39.53 for standard Pointer-generator + coverage)
- ROUGE-2: 17.82 (vs. 17.28)
- ROUGE-L: 37.30 (vs. 36.38)
- The model's performance remained highly stable across varying document lengths (tested between 1000 and 1400 words on the CNN dataset).
Limitations and Future Work
The authors acknowledge a few limitations in their current approach:
- Flattened Hierarchy: Serializing a parse tree via depth-first traversal flattens the data into a 1D sequence. While better than raw text, it still fails to capture the full hierarchical, multi-level structure of the parsing tree.
- Incomplete Gating: Currently, the dynamic selective gate is only applied to the word tokens in the source text. It ignores the parsing symbols during the gating process.
For future work, the authors suggest exploring better representation methods (like Graph Neural Networks or Tree-LSTMs) to model the true hierarchical structure of the parse trees. They also plan to integrate the parsing symbols into the dynamic gating mechanism to exert even finer control over the information flow.
Summary updated