Abstract
UniGLM is a novel foundation model designed to generate high-quality embeddings for Text-Attributed Graphs (TAGs) by training a single, unified language model across multiple domains simultaneously. By integrating an adaptive neighbor-sampling strategy with an efficient "lazy" memory bank, UniGLM outperforms specialized state-of-the-art models in node classification and link prediction. Furthermore, it demonstrates significant zero-shot transfer capabilities to unseen graph types, addressing the limitations of domain-specific fine-tuning.
Introduction
In real-world applications, graphs are frequently "text-attributed" (TAGs), where nodes are associated with textual information, such as citation networks (papers with titles/abstracts) or e-commerce graphs (products with descriptions). Traditionally, Pre-trained Language Models (PLMs) like BERT have been employed to transform this text into vector representations.
However, existing methodologies suffer from a "silo" problem: PLMs are typically fine-tuned for one specific graph at a time. This approach is computationally inefficient and fails to leverage the "scaling law" benefits observed in natural language processing. The primary challenge lies in the diverse structural properties (density vs. sparsity) and statistics of different graphs, which complicates the training of a single model capable of understanding the relationship between text and topology across varied domains.
Methodology: The UniGLM Framework
UniGLM (Unified Graph Language Model) introduces three primary innovations to bridge the gap between textual content and graph structure:
- Adaptive and Learnable Positive Scheme (AdaPS): Unlike standard contrastive learning that selects random neighbors as positive samples, UniGLM adaptively determines sampling strategies based on node degrees and graph density. It chooses between direct neighbors and higher-order (2-3 hop) neighbors, utilizing a learnable module guided by Personalized PageRank to weight the most informative neighbors.
- Domain-Aware Contrastive Learning: The model is trained to maximize similarity between an anchor node's text and its structurally similar neighbors across multiple datasets simultaneously, forcing the PLM to learn a universal mapping from text to graph position.
- Lazy Contrastive Module: To mitigate the high computational cost of encoding text for every neighbor, UniGLM employs a "lazy" dynamic embedding table (memory bank). By retrieving historical embeddings from previous batches instead of re-running the PLM, the framework significantly reduces GPU memory usage and accelerates training.
Experimental Setup
The model was rigorously evaluated on nine benchmark TAGs spanning two major domains:
- Academic Networks: Including
ogbn-arxivandChildren. - Amazon Product Networks: Including
Computers,History,Photo,Fitness,Products, andVideoGames.
The VideoGames dataset served to test in-domain transferability, while ogbn-arxiv was utilized for cross-domain transferability tests (e.g., training on product data and testing on research papers).
Key Results and Insights
UniGLM achieves state-of-the-art performance, ranking first in 27 out of 28 test cases for node classification across various graph backbones such as GCN and GraphSAGE.
Superior Transferability
In zero-shot settings, UniGLM significantly outperforms baselines like GIANT and PATTON. Notably, on the VideoGames dataset, it improved link prediction (Hits@100) by over 100% compared to the next best model. This suggests that underlying patterns in how text relates to graph structure transcend specific domains.
Efficiency and Joint Training
Despite its status as a foundation model, UniGLM is highly efficient. It completed training on seven datasets in just 13.5 hours, compared to the 5.4 days required by competitors like PATTON to train individual models. Furthermore, the "lazy" update module not only improved speed but also acted as a stabilizer, enhancing overall performance by preventing model deviation based on single mini-batches.
Limitations and Future Directions
While UniGLM represents a significant step forward, several areas remain for future exploration:
- Backbone Sensitivity: Performance varies with the base PLM (e.g., DeBERTa vs. Sentence-BERT), suggesting the need for an optimal base architecture for graph tasks.
- Scaling: Future work should aim to scale training to hundreds or thousands of diverse graphs to achieve true "foundation model" status.
- Task Variety: Beyond node classification and link prediction, extending UniGLM to graph-level classification or generative reasoning tasks is a promising direction.
Summary updated