Abstract
Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known. Our experiment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.
Introduction
Graph Attention Networks (GATs) have become a cornerstone of Graph Neural Networks (GNNs) by promising to focus on relevant neighbors while filtering out noise. However, standard GATs are trained implicitly; they lack direct supervision for attention weights, which often leads to inconsistent performance across different datasets, especially when graphs contain "noisy" edges between unrelated nodes. If a GAT attends to these mislinked neighbors, the resulting node representations suffer. This research addresses the fundamental gap in understanding how different attention forms (e.g., additive vs. dot-product) behave in noisy environments and proposes a self-supervised framework to ground attention in structural reality.
Methods: SuperGAT and Self-Supervision
The core innovation, SuperGAT, treats edge presence as a proxy for relational importance. By adding a secondary self-supervised task—predicting the existence of an edge between nodes $i$ and $j$ using their attention value—the model receives direct feedback on its "focus."
The researchers analyzed and extended two primary attention forms:
- GO (GAT Original): Uses a single-layer neural network for attention.
- DP (Dot-Product): Uses the dot product of node features.
From these, they derived two advanced variants:
- SuperGAT_SD (Scaled Dot-product): Scales the dot-product by $1/\sqrt{F}$ (where $F$ is the feature dimension) to prevent extreme attention values that might cause the model to collapse onto a single neighbor.
- SuperGAT_MX (Mixed): A hybrid approach that gates the GO attention with a sigmoid-activated DP attention, effectively filtering out unlikely neighbors.
Key Insights: The "Recipe" for Attention Design
A critical finding of this work is that the optimal attention mechanism is not universal but depends on the graph's structural properties: homophily (similarity between connected nodes) and average degree (graph density).
- GO vs. DP Trade-off: GO attention is superior at capturing "label-agreement" (homophily), while DP attention is significantly better at the self-supervised task of edge prediction.
- Design Recipe:
- In low-homophily settings, SuperGAT_SD is preferred as it excels at identifying sparse, relevant connections.
- In high-homophily settings with moderate density, SuperGAT_MX is most effective, combining the strengths of both additive and dot-product mechanisms.
Experimental Results
The authors validated SuperGAT across 17 real-world datasets and 144 synthetic graphs.
- Performance: SuperGAT outperformed original GAT and GCN baselines on 15 of 17 real-world datasets. Notable gains were seen in PubMed (81.7% accuracy vs. 79.0% for GAT) and PPI (74.4 micro-F1 vs. 72.2 for GAT).
- Robustness: By supervising attention with edge prediction, SuperGAT effectively learns to ignore noisy links, maintaining high performance where standard GATs degrade.
- Efficiency: The self-supervised task adds negligible overhead, maintaining the same time and space complexity as the original GAT. Training remains practical through efficient negative sampling techniques.
Conclusion
SuperGAT provides a robust framework for graph attention by linking it to the fundamental task of edge prediction. By understanding the interplay between attention forms and graph characteristics like homophily and density, researchers can now follow a principled "recipe" to design more effective GNNs for noisy, real-world data. Future work may explore integrating degree-normalization directly into attention to further improve performance on extremely dense graphs.
Summary updated