TL;DR
This paper introduces a novel abstractive summarization model that uses Graph Convolutional Networks (GCNs) to encode the grammatical structure of a document. By combining these syntactic features with traditional semantic embeddings through a selective attention gate, the model effectively filters out "noise" and focuses on the most salient information, achieving state-of-the-art results on the CNN/Daily Mail dataset.
What Problem Does This Paper Address?
Abstractive summarization—generating a concise summary in original words rather than just cutting and pasting sentences—is notoriously difficult. Standard Sequence-to-Sequence (Seq2Seq) models often struggle with two things:
- Saliency: Identifying which parts of a long document are actually important.
- Structure: Understanding the non-linear relationship between words (syntax).
While previous research has tried using "selective gates" to filter information, these gates usually only look at the surface-level meaning (semantics). They ignore the syntactic structure (how words relate grammatically), which is a powerful signal for determining what a sentence is actually about. Furthermore, modeling syntax across an entire document (multiple sentences) is computationally tricky because dependency trees are non-linear.
Methods
The authors propose a multi-layered encoder-decoder architecture with three specific components:
- Semantic Document Encoder: A standard bidirectional LSTM (BiLSTM) that processes the document as a sequence to capture the contextual meaning of words.
- Syntactic Document Encoder (The GCN): The authors build a document-level graph where nodes are words and edges represent dependency parsing relations. They link the root nodes of individual sentences to create a single, unified graph for the whole document. They then use stacked GCNs to learn representations that capture both local and long-distance grammatical relationships.
- Attention Information Gate: This is the "filter." It takes the combined semantic and syntactic representations and passes them through an attention mechanism to create a global document vector. This vector is used to calculate a "selective gate" for each word. If a word isn't deemed "salient" based on the global context, the gate suppresses its influence before it ever reaches the decoder.
The decoder uses a pointer-generator network (allowing it to copy rare words from the source) with a coverage mechanism to prevent the model from repeating itself.
Datasets
The model was evaluated on the CNN/Daily Mail dataset, a standard benchmark for long-text summarization.
- Training set: 287,277 pairs
- Validation set: 13,368 pairs
- Test set: 11,490 pairs
- Preprocessing: They used the Stanford Parser to generate dependency trees with 39 distinct edge labels to feed into the GCN.
Key Insights and Ideas
- Syntax as a Saliency Signal: The core "aha moment" is that grammar isn't just for correctness; it's a map of importance. By knowing which words are "heads" in a dependency tree, the model can better identify the primary subjects and actions of a document.
- Heterogeneous Graphs: Instead of treating all grammatical relations the same, the GCN uses separate parameters for different edge types and directions. This allows the model to distinguish between, for example, a subject-verb relationship and a modifier-noun relationship.
- Global Filtering: By using a global document vector to gate individual words, the model performs "information selection" early in the process. This reduces the burden on the decoder, which no longer has to wade through irrelevant details.
Key Results
The proposed model outperformed several strong baselines, including both extractive and abstractive methods.
| Method | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Lead-3 (Baseline) | 40.34 | 17.70 | 36.57 |
| Bottom-up (Previous SOTA) | 41.22 | 18.68 | 38.34 |
| Our Model | 41.79 | 19.06 | 38.56 |
Ablation studies confirmed the importance of the new components: removing the GCN caused a significant drop in performance (dropping ROUGE-1 from 32.39 to 31.46 on a subset of the data), proving that syntactic information provides a unique value that semantic LSTMs cannot capture alone.
Limitations and Future Work
- Parsing Dependency: The model relies on an external parser (Stanford Parser). If the parser makes a mistake, those errors propagate into the GCN.
- Computational Complexity: Building a document-wide graph and running multi-layer GCNs is more computationally expensive than standard RNN or Transformer approaches.
- Future Directions: The authors suggest that further refining how sentence-to-sentence connections are handled in the graph could improve the flow and coherence of the generated summaries. They also note that this syntactic approach could likely be integrated into Transformer-based architectures for even better performance.
Summary updated