Boosting Graph Foundation Model from Structural Perspective

Yao Cheng, Yige Zhao, Jianxiang Yu, Xiang Li

Abstract

Graph foundation models have recently attracted significant attention due to their strong generalizability. Although existing methods resort to language models to learn unified semantic representations across domains, they often disregard the unique structural characteristics of graphs from different domains. To address this problem, we propose BooG, a model that boosts graph foundation models from a structural perspective. BooG constructs virtual super nodes to unify structural characteristics of graph data from different domains. Specifically, these super nodes fuse the information of anchor nodes and class labels, where each anchor node captures the information of a node or a graph instance to be classified. Instead of using the raw graph structure, we connect super nodes to all nodes within their neighborhood by virtual edges. This new structure allows for effective information aggregation while unifying cross-domain structural characteristics. Additionally, we propose a novel pre-training objective based on contrastive learning, which learns more expressive representations for graph data and generalizes effectively to different domains and downstream tasks.

Introduction

In the world of machine learning, "Foundation Models" like GPT-4 have revolutionized how we handle text and images. However, creating a foundation model for graphs—data structures representing everything from social networks to molecular bonds—has proven much trickier. While Large Language Models (LLMs) have helped us understand the text attached to graph nodes, they often stumble when faced with the vastly different structures of different domains.

Most modern graph models for Text-Attributed Graphs (TAGs) use LLMs to create a unified semantic space. This works well for text, but graphs from different domains have "shapes" that do not translate well. For example, citation networks are highly clustered, whereas molecular graphs have rigid, specific topological structures based on chemical bonds. Existing models often struggle to generalize because they are too tied to these raw, domain-specific structures.

The BooG Framework

BooG shifts the focus from raw topology to a standardized structural perspective through three key innovations: sub-graph standardization, super nodes, and virtual edges.

The BooG framework consists of a self-supervised pre-training phase and an inference phase, utilizing a frozen LM and a trainable encoder.

Sub-graph Standardization

Every task (node, edge, or graph-level) is converted into a sub-graph centered on an "anchor node." This creates a consistent "template" for the model to process, regardless of the original data source.

Super Nodes and Virtual Edges

For every anchor node, the model creates "Super Nodes"—one for each possible class label. A Super Node is a virtual entity that combines the anchor node's features with the textual description of a specific class (e.g., "This node is a paper about Diabetes").

BooG unifies graph structures by creating super nodes that fuse anchor nodes with class labels, connected to neighbors via virtual edges.

Instead of relying on the messy raw edges of the graph, BooG establishes "Virtual Edges" between Super Nodes and the anchor's neighbors. It uses a learnable attention mechanism to aggregate information. This allows the model to treat classification as a series of hypothesis-driven scenarios: If this node belongs to Class X, which of its neighbors provide the most supporting evidence?

Contrastive Pre-training

The model is pre-trained using a self-supervised contrastive loss. It learns to pull the representation of a Super Node closer to its "correct" class label while pushing it away from "incorrect" class hypotheses. This approach makes the model much more robust for zero-shot learning, where it can understand textual descriptions of labels it hasn't seen during training.

Experimental Evaluation

The researchers evaluated BooG on seven diverse datasets across four distinct domains:

  • Citation Networks: Cora, Pubmed, ogbn-arxiv.
  • Web Links: Wiki-CS.
  • Product Networks: Ele-Fashion.
  • Molecular Graphs: PCBA, HIV.

Key Results

  • Superior Generalization: In zero-shot scenarios, BooG outperformed the previous state-of-the-art (UniGraph) by nearly 6% on the Cora dataset.
  • State-of-the-Art Performance: In supervised settings, BooG achieved the highest accuracy on almost every benchmark, notably hitting 88.51% on Pubmed.
  • Efficiency: BooG is remarkably efficient. Unlike generative models that require massive reconstruction, BooG uses a lightweight encoder and freezes the underlying LM. Pre-training on the Cora dataset took only 43 minutes, compared to 2.8 hours for UniGraph.

Conclusion

BooG demonstrates that raw graph topology can be a hindrance to cross-domain generalization. By "virtualizing" the aggregation process through super nodes and virtual edges, the model learns a universal way of gathering neighborhood information. While it relies on high-quality textual descriptions of class labels, its efficiency and performance across zero-shot and supervised tasks mark a significant step forward for graph foundation models.

Code and data: https://anonymous.4open.science/r/BooG-EE42/

Summary updated

Abstract

Graph foundation models have recently attracted significant attention due to its strong generalizability. Although existing methods resort to language models to learn unified semantic representations across domains, they disregard the unique structural characteristics of graphs from different domains. To address the problem, in this paper, we boost graph foundation model from structural perspective and propose BooG. The model constructs virtual super nodes to unify structural characteristics of graph data from different domains. Specifically, the super nodes fuse the information of anchor nodes and class labels, where each anchor node captures the information of a node or a graph instance to be classified. Instead of using the raw graph structure, we connect super nodes to all nodes within their neighborhood by virtual edges. This new structure allows for effective information aggregation while unifying cross-domain structural characteristics. Additionally, we propose a novel pre-training objective based on contrastive learning, which learns more expressive representations for graph data and generalizes effectively to different domains and downstream tasks. Experimental results on various datasets and tasks demonstrate the superior performance of BooG. We provide our code and data here: https://anonymous.4open.science/r/BooG-EE42/.
BibTeX
@misc{cheng2024boosting,
  author = {Yao Cheng and Yige Zhao and Jianxiang Yu and Xiang Li},
  title = {{Boosting Graph Foundation Model from Structural Perspective}},
  year = {2024},
  eprint = {2407.19941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.19941},
}
Figure