TL;DR
This paper introduces MDGFM, a novel framework for multi-domain graph foundation models that tackles the challenge of differing graph topologies. By aligning both features and graph structures across domains, MDGFM achieves more robust and effective knowledge transfer to unseen graph datasets, outperforming existing methods in few-shot learning scenarios.
What Problem Does This Paper Address?
The field of graph machine learning is increasingly looking towards "foundation models" – models pre-trained on vast amounts of data to be adaptable to various downstream tasks. However, a major hurdle in applying this to graphs is the significant diversity in graph structures (topologies) across different domains (e.g., social networks vs. citation networks). Existing methods often struggle with these topological differences, leading to poor generalization. Furthermore, real-world graphs are often noisy and susceptible to adversarial attacks, further complicating robust knowledge transfer. This paper aims to bridge this gap by developing a foundation model that can effectively learn from and transfer knowledge across diverse graph domains, even when faced with noisy data and structural variations.
Methods
The core of MDGFM lies in its Multi-domain pre-training module and Downstream target-domain adaptation module.
Multi-domain pre-training: * Feature Projection into Unified Semantic Space: Initially, node features from different domains are projected into a common dimensionality using PCA. Then, domain-specific tokens and a shared "global" token are used to align features semantically, allowing the model to capture both domain-specific nuances and universal patterns. * Graph Topology-aware Alignment: This is a key innovation. Instead of just relying on existing graph structures, MDGFM employs Graph Structure Learning (GSL) to refine the graph topology. It adaptively balances node features and structural information using a "balance token" and refines the adjacency matrix to be more robust and aligned across domains. This process aims to eliminate noise and create a more consistent structural representation. * Contrastive Learning: The pre-training objective uses a contrastive loss that maximizes mutual information between original and refined graph representations, encouraging the learning of domain-invariant knowledge.
Downstream target-domain adaptation: * Dual-stream Prompt Framework: This module uses a "pre-training & prompting" paradigm. It employs a "meta prompt" to adjust learned knowledge based on relationships between target and source domains, and a "specific prompt" for direct alignment with the target domain using limited samples. This approach is inspired by in-context learning and aims to efficiently adapt the pre-trained model.
Datasets
The paper uses a variety of datasets to demonstrate its capabilities:
- Homophilic Citation Networks: Cora, Citeseer, Pubmed
- Heterophilic Web Graphs: Cornell, Chameleon, Squirrel
- Large-scale Social Network: Penn94 (used as a target domain for evaluating generalization)
These datasets represent diverse graph structures and characteristics, allowing for a comprehensive evaluation of the model's cross-domain transfer capabilities.
Key Insights and Ideas
- Topology is King (for Transfer): The paper strongly emphasizes that simply aligning node features isn't enough for robust cross-domain graph learning. Aligning and refining the graph topology itself is crucial for effective knowledge transfer, especially when dealing with diverse graph structures.
- Adaptive Feature & Topology Balancing: The introduction of a "balance token" to dynamically weigh node features and structural information during GSL is a novel way to handle the interplay between these two crucial aspects of graph data.
- Domain-Invariant Learning is Essential: The success of the "shared token" in aligning features across domains highlights the importance of identifying and leveraging commonalities, rather than just focusing on domain-specific differences.
- Robustness to Noise and Attacks: The proposed topology alignment mechanism inherently makes the model more resilient to noisy edges and adversarial perturbations, a significant advantage for real-world applications.
Key Results
- Superior Few-Shot Performance: MDGFM consistently outperforms existing supervised, graph pre-training, and graph prompting baselines in both one-shot and few-shot node classification tasks across homophilic and heterophilic datasets. For instance, it achieves up to an 18.04% improvement over the second-best model in the one-shot scenario.
- Robustness: The model demonstrates strong robustness against edge addition/deletion attacks and meta-attacks, significantly outperforming competitors like GCOPE and MDGPT, which show performance degradation under such conditions.
- Domain Sensitivity: MDGFM shows greater stability and less performance degradation when source domains are removed compared to baselines, indicating it doesn't rely heavily on specific source domains.
- Low Sensitivity to Hyperparameters: The model exhibits low sensitivity to the
khyperparameter in its kNN-based graph structure learning, making it easier to tune.
Limitations and Future Work
- Computational Cost: While the paper mentions efficiency improvements with kNN, the overall pre-training process for multiple domains can still be computationally intensive.
- Dynamic Graphs: The current framework is designed for static graphs. The authors suggest future work could extend their approach to dynamic graphs, which evolve over time.
- Heterogeneous Graphs: While the paper addresses homophilic and heterophilic graphs, extending the framework to handle truly heterogeneous graphs (with different node and edge types) is a potential future direction.
- Theoretical Guarantees: The paper provides theoretical analysis for domain generalization, but further exploration of the theoretical underpinnings of topology alignment could be beneficial.
Summary updated