Selective Attention Encoders by Syntactic Graph Convolutional Networks for Document Summarization

Haiyang Xu, Yun Wang, Kun Han, Baochang Ma, Junwen Chen, Xiangang Li

Abstract

Abstractive text summarization is a challenging task, and one need to design a mechanism to effectively extract salient information from the source text and then generate a summary. A parsing process of the source text contains critical syntactic or semantic structures, which is useful to generate more accurate summary. However, modeling a parsing tree for text summarization is not trivial due to its non-linear structure and it is harder to deal with a document that includes multiple sentences and their parsing trees. In this paper, we propose to use a graph to connect the parsing trees from the sentences in a document and utilize the stacked graph convolutional networks (GCNs) to learn the syntactic representation for a document. The selective attention mechanism is used to extract salient information in semantic and structural aspect and generate an abstractive summary. We evaluate our approach on the CNN/Daily Mail text summarization dataset. The experimental results show that the proposed GCNs based selective attention approach outperforms the baselines and achieves the state-of-the-art performance on the dataset.

Introduction

Abstractive summarization aims to generate concise summaries using original phrasing, a task significantly more complex than extractive methods that simply select existing sentences. Standard Sequence-to-Sequence (Seq2Seq) models often struggle with identifying salient information in long documents and capturing the non-linear syntactic relationships between words. While "selective gates" have been used to filter information, they typically focus on surface-level semantics, ignoring the rich structural signals provided by dependency parsing. This paper addresses these challenges by integrating syntactic structure into the encoding process using Graph Convolutional Networks (GCNs).

Methods

The proposed architecture consists of a multi-layered encoder-decoder framework designed to fuse semantic and syntactic information.

The proposed model architecture featuring GCNs and a selective attention gate.

Semantic and Syntactic Encoding

The model employs a dual-encoder strategy:

  • Semantic Document Encoder: A bidirectional LSTM (BiLSTM) captures the sequential contextual meaning of words.
  • Syntactic Document Encoder (GCN): To model document-level syntax, the authors construct a graph where nodes represent words and edges represent dependency parsing relations. Sentence-level dependency trees are unified into a single document graph by linking root nodes. Stacked GCNs then process this graph to learn representations capturing both local and long-distance grammatical relationships.

Selective Attention Information Gate

The "Attention Information Gate" acts as a filter. It combines semantic and syntactic representations to generate a global document vector. This vector determines a "selective gate" for each word, suppressing non-salient information before it reaches the decoder. The decoder itself is a pointer-generator network with a coverage mechanism to handle rare words and prevent repetition.

Key Results

The model was evaluated on the CNN/Daily Mail dataset, utilizing the Stanford Parser for dependency tree generation.

Comparison of the proposed model against state-of-the-art extractive and abstractive baselines on the CNN/Daily Mail dataset.

As shown in the results table, the proposed model achieves state-of-the-art performance, outperforming strong baselines like Bottom-up and Lead-3 across all ROUGE metrics. Ablation studies further validated the design: removing the GCN component led to a significant performance drop (e.g., ROUGE-1 decreasing from 32.39 to 31.46 on a subset), confirming that syntactic information provides unique value beyond semantic LSTMs.

Conclusion

This work demonstrates that syntactic structure is a powerful signal for determining saliency in document summarization. By using GCNs to encode document-level dependency graphs and a selective attention gate to filter information, the model effectively identifies the primary subjects and actions of a text. Future work may involve refining sentence-to-sentence graph connections and integrating this syntactic approach into Transformer-based architectures to further improve coherence and computational efficiency.

Summary updated

Abstract

Abstractive text summarization is a challenging task, and one need to design a mechanism to effectively extract salient information from the source text and then generate a summary. A parsing process of the source text contains critical syntactic or semantic structures, which is useful to generate more accurate summary. However, modeling a parsing tree for text summarization is not trivial due to its non-linear structure and it is harder to deal with a document that includes multiple sentences and their parsing trees. In this paper, we propose to use a graph to connect the parsing trees from the sentences in a document and utilize the stacked graph convolutional networks (GCNs) to learn the syntactic representation for a document. The selective attention mechanism is used to extract salient information in semantic and structural aspect and generate an abstractive summary. We evaluate our approach on the CNN/Daily Mail text summarization dataset. The experimental results show that the proposed GCNs based selective attention approach outperforms the baselines and achieves the state-of-the-art performance on the dataset.
BibTeX
@inproceedings{xu2020selective,
  author = {Haiyang Xu and Yun Wang and Kun Han and Baochang Ma and Junwen Chen and Xiangang Li},
  title = {{Selective Attention Encoders by Syntactic Graph Convolutional Networks for Document Summarization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=X6Z5vZm8em},
}
Figure