ISHNE: Influence Self-attention for Heterogeneous Network Embedding

Yan Yang, Qiuyan WangarXiv 2021 · DOI · arXiv

Abstract

In recent years, Graph Neural Networks have received enormous attention for their potential in modeling network traits such as macrostructure and single node attributes. However, prior works mainly focus on homogeneous networks and lack the capacity to characterize network heterogeneity. Furthermore, most existing literature cannot model influence at a microscopic level, making it difficult to capture the joint relations between heterogeneity and mutual interactions across multiple relation types. We propose ISHNE (Influence Self-attention network), which redesigns the attention mechanism with an influence factor at the single-type relation level. This allows the model to learn importance coefficients from adjacent neighbors under meta-path based patterns. To incorporate heterogeneous meta-paths into a unified dimension, we developed a self-attention framework for meta-path relation fusion. Experimental results demonstrate that ISHNE outperforms state-of-the-art baselines and provides a promising approach for depicting complex heterogeneous interactive relations.

Introduction: The Challenge of Heterogeneity

Most real-world networks—such as academic citation graphs or movie databases—are heterogeneous, containing diverse node types (e.g., authors, papers, subjects) and complex relationships. While standard Graph Neural Networks (GNNs) excel in homogeneous settings, they often overlook the "microscopic" nuances of heterogeneous data.

Specifically, two major gaps exist in current research:

  1. Hidden Interactions: Standard models often treat connections as simple links, ignoring the inherent "social influence" or asymmetric impact one node may exert over another. For instance, a senior researcher typically has a different influence on a junior co-author than vice versa.
  2. Meta-path Integration: Effectively fusing information from various "meta-paths" (sequences of relations like Author-Paper-Author) into a single, cohesive node representation remains a significant challenge.

Methodology: The ISHNE Framework

ISHNE addresses these gaps through a two-level hierarchical attention architecture designed to capture both local neighborhood influence and global semantic relations.

The ISHNE architecture featuring hierarchical attention and influence mechanisms.

1. Meta-Path Based Neighborhood Influence Attention

Unlike standard Graph Attention Networks (GATs) that often assume symmetric weights, ISHNE introduces an explicit influence factor. This factor is projected into the attention mechanism, allowing the model to learn asymmetric importance coefficients. By accounting for the "influence" of a node, the model can more accurately represent how a "superstar" node impacts its neighbors differently than a standard node would.

2. Semantic Relation Self-Attention

Nodes in a heterogeneous network often participate in multiple meta-paths (e.g., a movie linked to both actors and directors). ISHNE employs a second level of self-attention to fuse these distinct semantic embeddings. This fusion step automatically learns the relative importance of different meta-paths for a given task, ensuring a unified and robust node representation.

3. Multi-head Attention

To stabilize learning and capture a diverse range of features, the neighborhood attention is implemented using multiple "heads," a technique proven effective in modern transformer-based architectures.

Experimental Evaluation

The model was evaluated on two benchmark heterogeneous information networks (HINs):

  • ACM Dataset: An academic network (3,025 papers, 5,835 authors, 56 subjects) using meta-paths like Paper-Author-Paper (PAP) and Paper-Subject-Paper (PSP).
  • IMDB Dataset: A movie network (4,780 movies, 5,841 actors, 2,269 directors) utilizing meta-paths like Movie-Actor-Movie (MAM) and Movie-Director-Movie (MDM).
Performance comparison of ISHNE against baselines on ACM and IMDB datasets.

As shown in the results, ISHNE consistently outperformed state-of-the-art baselines like GCN, GAT, and HAN:

  • ACM Performance: ISHNE achieved a Micro-F1 of 83.86%, surpassing HAN's 82.26%.
  • IMDB Performance: The performance gap was more pronounced on the IMDB dataset, with ISHNE reaching a Micro-F1 of 53.44% compared to HAN's 50.58%. This suggests that ISHNE is particularly adept at mining rich latent relations in complex datasets.
  • Robustness: Similar leads were observed in Macro-F1 scores, confirming the model's robustness across different class distributions.

Discussion and Future Directions

The success of ISHNE highlights that asymmetry matters in heterogeneous networks. By explicitly modeling influence, the framework moves closer to capturing real-world social and professional dynamics. Furthermore, the hierarchical approach maintains the distinct "flavors" of different connection types before final fusion.

Limitations and Future Work

Despite its strong performance, several areas for improvement remain:

  • Scalability: The computational cost of calculating multiple meta-paths and hierarchical attention may limit scalability to massive, billion-node graphs.
  • Automated Discovery: Currently, the model relies on pre-defined meta-paths. Future research could explore automated meta-path discovery to reduce the need for domain-specific expertise.
  • Interpretability: While attention weights provide some insight, a more qualitative analysis of which specific nodes are deemed "influential" would further validate the model's underlying logic.

Summary updated

Abstract

In recent years, Graph Neural Networks has received enormous attention from academia for its huge potential of modeling the network traits such as macrostructure and single node attributes. However, prior mainstream works mainly focus on homogeneous network and lack the capacity to characterize the network heterogeneous property. Besides, most previous literature cannot model the influence under microscope vision, making it infeasible to model the joint relation between the heterogeneity and mutual interaction within multiple relation type. In this paper, we propose an Influence Self-attention network to address the difficulties mentioned above. To model heterogeneity and mutual interaction, we redesign attention mechanism with influence factor on the single-type relation level, which learns the importance coefficient from its adjacent neighbors under the same meta-path based patterns. To incorporate the heterogeneous meta-path in a unified dimension, we developed a self-attention based framework for meta-path relation fusion according to the learned meta-path coefficient. Our experimental results demonstrate that our framework not only achieve higher results than current state-of-the-art baselines, but also show promising vision on depicting heterogeneous interactive relations under complicated network structure.
BibTeX
@misc{yang2021ishne,
  author = {Yan Yang and Qiuyan Wang},
  title = {{ISHNE: Influence Self-attention for Heterogeneous Network Embedding}},
  year = {2021},
  eprint = {2103.00118},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2103.00118},
}
Figure