Edge Attention-based Multi-Relational Graph Convolutional Networks

Jinbo BiarXiv 2018 · arXiv

TL;DR

This paper introduces EAGCN, a novel graph convolutional network that leverages edge attributes to better represent molecular structures. By learning attention weights for different types of chemical bonds, EAGCN achieves state-of-the-art performance in predicting molecular properties, offering a more nuanced understanding of atomic interactions.

What Problem Does This Paper Address?

Traditional Graph Convolutional Networks (GCNs) often treat all connections (edges) in a graph equally, typically using a simple binary adjacency matrix. However, in domains like chemistry, the relationships between nodes (atoms) are far more complex and can be characterized by multiple attributes (e.g., bond order, aromaticity, ring status). These attributes significantly influence the overall properties of a molecule. Existing methods struggle to effectively incorporate this rich relational information, leading to a potential loss of crucial predictive power. This paper aims to bridge this gap by developing a GCN that can dynamically learn and utilize these multi-relational edge attributes for improved graph representation and property prediction.

Methods

The core innovation is the Edge Attention Layer. Instead of a single adjacency matrix, EAGCN constructs a set of weighted adjacency matrices, one for each edge attribute. These weights are learned through a dictionary that maps each possible value of an edge attribute to a specific attention strength. This dictionary is shared across all molecules for a given attribute, allowing the model to learn generalizable patterns of interaction.

Key aspects of the method:

  • Multi-Relational Graphs: The model explicitly handles graphs where edges have multiple types of attributes.
  • Edge Attention: For each edge attribute, a learnable attention weight is assigned to represent the strength of interaction between connected nodes.
  • Dictionary Learning: A dictionary stores these attention weights for each edge attribute, enabling the model to learn a fixed set of interaction strengths for each attribute type across the entire dataset.
  • Weighted Adjacency Tensor: The learned dictionaries are used to construct a tensor of weighted adjacency matrices, capturing the multi-relational nature of the graph.
  • Aggregation: Node features are aggregated based on these weighted adjacency matrices, allowing for a more informed representation. The paper explores two aggregation strategies: concatenation of features from different attribute layers and a weighted sum.

Datasets

The proposed EAGCN model was evaluated on four benchmark datasets commonly used in molecular machine learning:

  • Tox21: A dataset for predicting biological outcomes of compounds across 12 pathway assays.
  • HIV: A dataset for predicting the ability of compounds to inhibit HIV replication.
  • Freesolv: A dataset containing experimental and calculated hydration free energies for small molecules.
  • Lipophilicity (Lipo): A dataset providing octanol/water distribution coefficients, an important measure of a molecule's interaction with solvents.

These datasets were downloaded from the MoleculeNet website.

Key Insights and Ideas

  • Edge Attributes are Crucial: The paper strongly suggests that explicitly modeling the diverse attributes of chemical bonds is key to unlocking better molecular property prediction.
  • Shared Attention for Generalization: By sharing edge attention dictionaries across molecules, EAGCN learns invariant features that capture general atomic interaction patterns, making it robust to variations in molecular structure and size.
  • Interpretability through Dictionaries: The learned edge attention dictionaries provide insights into which types of atomic interactions are most important for specific properties or datasets, offering a degree of interpretability. For instance, the interaction between Carbon and Tin (C-Sn) was found to be important for toxicity prediction (Tox21) but not for HIV inhibition.
  • Handling Varying Graph Sizes: The edge attention mechanism, by focusing on attribute-based weights rather than individual edges, makes the model inherently robust to the varying sizes of molecular graphs.

Key Results

EAGCN demonstrated superior performance across multiple datasets and tasks:

  • Classification: EAGCN achieved state-of-the-art or near state-of-the-art results on the Tox21 and HIV datasets, outperforming baselines like GCN and Weave. For example, it improved upon GCN by 2.4% on Tox21 test data and 7.8% on HIV test data.
  • Regression: On the Freesolv and Lipophilicity datasets, EAGCN achieved significant improvements over existing methods, reaching an RMSE of 0.61 for Lipophilicity, which was the best reported result.
  • Robustness: The model showed smaller gaps between training and validation/test scores, indicating good generalization.
  • Visualization: Heatmaps of the learned edge attention dictionaries revealed that similar attention types across different datasets often had similar weight values, suggesting the learning of general atomic interaction patterns.

Limitations and Future Work

  • Limitations: The paper acknowledges that while the model performs well, the interpretation of attention weights is still an area for further exploration. The current approach focuses on intra-molecular interactions, and extending it to inter-molecular interactions could be beneficial.
  • Future Work: The authors suggest extending the EAGCN model to handle multiple types of data in different situations, implying its potential applicability beyond molecular property prediction. They also propose further investigation into the interpretability of the learned attention mechanisms.

Summary updated

Abstract

Graph convolutional network (GCN) is generalization of convolutional neural network (CNN) to work with arbitrarily structured graphs. A binary adjacency matrix is commonly used in training a GCN. Recently, the attention mechanism allows the network to learn a dynamic and adaptive aggregation of the neighborhood. We propose a new GCN model on the graphs where edges are characterized in multiple views or precisely in terms of multiple relationships. For instance, in chemical graph theory, compound structures are often represented by the hydrogen-depleted molecular graph where nodes correspond to atoms and edges correspond to chemical bonds. Multiple attributes can be important to characterize chemical bonds, such as atom pair (the types of atoms that a bond connects), aromaticity, and whether a bond is in a ring. The different attributes lead to different graph representations for the same molecule. There is growing interests in both chemistry and machine learning fields to directly learn molecular properties of compounds from the molecular graph, instead of from fingerprints predefined by chemists. The proposed GCN model, which we call edge attention-based multi-relational GCN (EAGCN), jointly learns attention weights and node features in graph convolution. For each bond attribute, a real-valued attention matrix is used to replace the binary adjacency matrix. By designing a dictionary for the edge attention, and forming the attention matrix of each molecule by looking up the dictionary, the EAGCN exploits correspondence between bonds in different molecules. The prediction of compound properties is based on the aggregated node features, which is independent of the varying molecule (graph) size. We demonstrate the efficacy of the EAGCN on multiple chemical datasets: Tox21, HIV, Freesolv, and Lipophilicity, and interpret the resultant attention weights.
BibTeX
@misc{bi2018edge,
  author = {Jinbo Bi},
  title = {{Edge Attention-based Multi-Relational Graph Convolutional Networks}},
  year = {2018},
  eprint = {1802.04944},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.04944},
}
Figure