Attention Guided Graph Convolutional Networks for Relation Extraction

Zhijiang Guo, Yan Zhang, Wei Lu

Abstract

Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our model can be understood as a soft-pruning approach that automatically learns how to selectively attend to the relevant sub-structures useful for the relation extraction task. Extensive results on various tasks including cross-sentence n-ary relation extraction and large-scale sentence-level relation extraction show that our model is able to better leverage the structural information of the full dependency trees, giving significantly better results than previous approaches.

The Challenge of Dependency Trees

Relation Extraction (RE) is the task of identifying how entities in a text are related—for instance, determining if a specific gene mutation is "sensitive" to a particular drug. To achieve this, models often utilize dependency trees to map the grammatical structure of a sentence.

However, dependency trees are inherently noisy. A common strategy to mitigate this is "hard pruning," which involves cutting the tree down to the Shortest Dependency Path (SDP) between entities or the subtree under their Lowest Common Ancestor (LCA). While this removes noise, it often inadvertently deletes crucial context.

Example dependency tree where critical context ('partial response') lies outside the direct path between entities.

As shown in the figure above, in medical texts, a phrase like "partial response" might be located outside the standard pruned tree, causing a model to miss the relation entirely. The core challenge lies in retaining the useful structural information of the full tree without being overwhelmed by irrelevant data.

Methodology: Attention Guided Graph Convolutional Networks

The proposed AGGCN model treats the dependency tree as a starting point but moves beyond rigid pruning. The architecture consists of three main components designed to perform "soft pruning":

  1. Attention Guided Layer: Instead of deleting edges, the model uses a multi-head self-attention mechanism to transform the dependency tree into a fully connected, edge-weighted graph. Each "head" learns different weights, allowing the model to attend to various paths and sub-structures relevant to the specific relation.
  2. Densely Connected Layer: Standard GCNs often struggle with depth. AGGCN incorporates dense connections where each sub-layer receives the concatenated outputs of all preceding layers. This allows the model to capture both local and non-local interactions across the graph more effectively.
  3. Linear Combination Layer: This integrates the representations from the various attention heads into a single hidden representation.
The AGGCN architecture, featuring Attention Guided Layers and Densely Connected Layers.

For sentence-level tasks where word order is vital, the authors also propose C-AGGCN, which adds a BiLSTM layer before the AGGCN to capture contextualized word sequences.

Key Insights and Results

The model was evaluated on three major benchmarks: cross-sentence n-ary relation extraction (PubMed), large-scale sentence-level extraction (TACRED), and the public SemEval-2010 Task 8 dataset.

  • Soft Pruning Superiority: Rule-based pruning is often a bottleneck. By allowing the model to learn its own pruning strategy through attention, it recovers information that standard models discard. Unlike previous GCN-based models that performed best with limited pruning distances, AGGCN performs best when given the full dependency tree, demonstrating high robustness to noise.
  • Performance Gains: AGGCN outperformed the previous state-of-the-art (Graph State LSTM) by 8% on ternary tasks and 6% on binary tasks. On TACRED, the contextualized C-AGGCN achieved an F1 score of 69.0, significantly improving over the standard C-GCN (66.4).
  • Data Efficiency: C-AGGCN demonstrated superior learning efficiency; when trained on only 80% of the TACRED dataset, it outperformed a standard C-GCN trained on the full 100%.

Conclusion

AGGCN demonstrates that "soft pruning" via attention is a more effective way to leverage dependency structures than traditional rule-based methods. By combining multi-head attention with dense connections, the model successfully navigates the noise of full dependency trees to achieve state-of-the-art results. Future work may involve integrating these learned graph representations with large-scale pre-trained language models to further enhance extraction accuracy.

Summary updated

Abstract

Dependency trees convey rich structural information that is proven useful for extracting relations among entities in text. However, how to effectively make use of relevant information while ignoring irrelevant information from the dependency trees remains a challenging research question. Existing approaches employing rule based hard-pruning strategies for selecting relevant partial dependency structures may not always yield optimal results. In this work, we propose Attention Guided Graph Convolutional Networks (AGGCNs), a novel model which directly takes full dependency trees as inputs. Our model can be understood as a soft-pruning approach that automatically learns how to selectively attend to the relevant sub-structures useful for the relation extraction task. Extensive results on various tasks including cross-sentence n-ary relation extraction and large-scale sentence-level relation extraction show that our model is able to better leverage the structural information of the full dependency trees, giving significantly better results than previous approaches.
BibTeX
@inproceedings{guo2019attention,
  author = {Zhijiang Guo and Yan Zhang and Wei Lu},
  title = {{Attention Guided Graph Convolutional Networks for Relation Extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=14z6m0HEDKb},
}
Figure