SlotGAT: Slot-based Message Passing for Heterogeneous Graph Neural Network

Ziang Zhou, Jieming Shi, Renchi Yang, Yuanhang Zou, Qing Li

Abstract

Heterogeneous graphs are ubiquitous to model complex data. There are urgent needs on powerful heterogeneous graph neural networks to effectively support important applications. We identify a potential semantic mixing issue in existing message passing processes, where the representations of the neighbors of a node $v$ are forced to be transformed to the feature space of $v$ for aggregation, though the neighbors are in different types. That is, the semantics in different node types are entangled together into node $v$'s representation. To address the issue, we propose SlotGAT with separate message passing processes in slots, one for each node type, to maintain the representations in their own node-type feature spaces. Moreover, in a slot-based message passing layer, we design an attention mechanism for effective slot-wise message aggregation. Further, we develop a slot attention technique after the last layer of SlotGAT, to learn the importance of different slots in downstream tasks. Our analysis indicates that the slots in SlotGAT can preserve different semantics in various feature spaces. The superiority of SlotGAT is evaluated against 13 baselines on 6 datasets for node classification and link prediction. Our code is at https://github.com/scottjiao/SlotGAT_ICML23/.

Introduction: The Challenge of Semantic Mixing

Heterogeneous graphs contain multiple types of nodes and edges, representing complex real-world relationships. Existing Heterogeneous Graph Neural Networks (HGNNs) typically aggregate information from neighbors by transforming their features into the target node's feature space. However, this process often leads to semantic mixing. When a node (e.g., a "Paper") aggregates information from different types of neighbors (e.g., "Authors" and "Venues"), their distinct semantic signals are entangled into a single vector too early, leading to a loss of fine-grained information.

Comparison between standard message passing and SlotGAT's slot-based approach.

Methodology: Slot-based Message Passing

To address semantic mixing, SlotGAT introduces a "slot" mechanism where each node maintains separate representations for each node type in the graph. This ensures that information from different types remains distinct throughout the intermediate layers of the network.

The overall architecture of SlotGAT, illustrating the flow from slot initialization to downstream processing.

The model operates through four primary stages:

  1. Slot Initialization: Every node is assigned a set of slots corresponding to the number of node types. A node only initializes the slot matching its own type with its features; other slots are set to zero.
  2. Slot-wise Message Passing: Messages are passed between corresponding slots (e.g., an "Author" neighbor updates the "Author" slot of a "Paper" node). This preserves the "purity" of type-specific semantics across layers.
  3. Attention-based Aggregation: Within each layer, a self-attention mechanism determines the importance of neighbors. Crucially, the same attention weight is applied across all slots for a given neighbor-target pair to maintain consistency.
  4. Slot Attention (Final Layer): After multiple layers of message passing, a final attention module learns the relative importance of each slot for the specific downstream task, allowing the model to prioritize the most relevant semantic signals.

Experimental Evaluation

The authors evaluated SlotGAT against 13 state-of-the-art baselines across six benchmark datasets, covering node classification (DBLP, IMDB, ACM, Freebase, PubMed) and link prediction (LastFM, PubMed).

Key Results

SlotGAT consistently outperformed competitors like HAN, MAGNN, and HGT. For instance, on the IMDB dataset, SlotGAT achieved a Micro-F1 of 68.54%, surpassing the best baseline (67.48%). On the large-scale Freebase dataset, it reached a Macro-F1 of 49.68%. In link prediction tasks on LastFM, SlotGAT achieved an ROC-AUC of 70.33%, representing a 2.74% improvement over simpleHGN. Paired t-tests confirmed that these improvements were statistically significant ($p < 0.05$).

Discussion and Insights

  • Semantic Preservation: Theoretical analysis confirms that slots capture type-specific global semantics, effectively acting as a structured filing system for graph information.
  • Explainability: The final slot attention weights ($\beta_t$) provide interpretability, showing which node types are most influential for specific predictions.
  • Efficiency: Despite the added complexity of multiple slots, SlotGAT remains computationally competitive with advanced models like HGT in terms of training time and memory usage.

Limitations and Future Work

The linear scaling of complexity with the number of node types may pose challenges for graphs with hundreds of types. Future research could explore selective slot assignment or dynamic dimensionality to enhance scalability for massive industrial knowledge graphs. Additionally, extending this slot-based approach to dynamic, evolving graphs remains an open challenge.

Summary updated

Abstract

Heterogeneous graphs are ubiquitous to model complex data. There are urgent needs on powerful heterogeneous graph neural networks to effectively support important applications. We identify a potential semantic mixing issue in existing message passing processes, where the representations of the neighbors of a node $v$ are forced to be transformed to the feature space of $v$ for aggregation, though the neighbors are in different types. That is, the semantics in different node types are entangled together into node $v$'s representation. To address the issue, we propose SlotGAT with separate message passing processes in slots, one for each node type, to maintain the representations in their own node-type feature spaces. Moreover, in a slot-based message passing layer, we design an attention mechanism for effective slot-wise message aggregation. Further, we develop a slot attention technique after the last layer of SlotGAT, to learn the importance of different slots in downstream tasks. Our analysis indicates that the slots in SlotGAT can preserve different semantics in various feature spaces. The superiority of SlotGAT is evaluated against 13 baselines on 6 datasets for node classification and link prediction. Our code is at https://github.com/scottjiao/SlotGAT_ICML23/.
BibTeX
@inproceedings{zhou2023slotgat,
  author = {Ziang Zhou and Jieming Shi and Renchi Yang and Yuanhang Zou and Qing Li},
  title = {{SlotGAT: Slot-based Message Passing for Heterogeneous Graph Neural Network}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=G6L1kwy9AA},
}
Figure