TL;DR
UniGraphLM is a unified graph language model that successfully bridges the gap between diverse graph structures and large language models (LLMs) across multiple domains and tasks. By combining a multi-scale graph neural network (GNN) encoder with a two-stage training strategy—graph-text contrastive pretraining and difficulty-aware curriculum alignment tuning—it achieves state-of-the-art generalization on node, edge, and graph-level tasks.
What Problem Does This Paper Address?
Graph Language Models (GLMs) aim to supercharge LLMs with the structural processing power of Graph Neural Networks (GNNs). Typically, this involves a two-stage pipeline: a GNN encodes the graph, and a projector maps these encodings into tokens the LLM can understand.
However, existing GLMs are heavily siloed. They are usually designed for a single domain (e.g., citation networks) or a single task type (e.g., node classification). When forced to handle diverse graph data, they fail due to massive variations in graph structures, feature distributions, and task granularities. Furthermore, different graph datasets exhibit varying degrees of compatibility with an LLM's token space, making a standard, one-size-fits-all alignment strategy highly suboptimal.
Methods
The authors introduce UniGraphLM, which tackles these challenges using a two-stage framework:
-
Stage 1: Graph-Text Pair Pretraining
To learn generalizable, text-aligned representations, the authors use an LLM (Qwen3-8B) to generate natural language descriptions of graph structures and attributes. A multi-scale GNN (GraphSAGE) is then trained alongside a text encoder using a novel Domain-aware Reweighted Contrastive Loss ($L_{DR-CLIP}$). This loss assigns higher weights to negative samples from distant domains, pushing different domains apart in the embedding space while preserving fine-grained, intra-domain semantics. Crucially, task-specific aggregation functions allow the same GNN to output node-, edge-, or graph-level representations. -
Stage 2: Curriculum Alignment Tuning
During instruction tuning, the GNN and LLM are frozen, and only the projector layer is updated. Because different datasets present different alignment difficulties, UniGraphLM introduces an online Difficulty Estimator. It tracks the gradient norm of the projector for each domain during training; a larger gradient norm implies poorer alignment. A scheduler then uses a temperature-scaled softmax to dynamically assign higher training weights to these harder domains.
Datasets
The model was evaluated across a diverse suite of eight benchmark datasets spanning three granularities:
- Node-level classification: Cora, PubMed, Arxiv (citation networks), and Wiki-CS (web hyperlink network).
- Edge-level classification: WN18RR and FB15K237 (knowledge graphs).
- Graph-level classification: ChemPCBA and ChemHIV (molecular datasets).
Key Insights and Ideas
- Unified Multi-Scale Encoding: Instead of training separate GNNs, UniGraphLM uses a single GNN with task-specific pooling and aggregation layers. By concatenating local node embeddings with global graph context, the model seamlessly handles node, edge, and graph-level tasks simultaneously.
- Not All Negatives Are Equal: In multi-domain contrastive learning, treating all negative samples uniformly hurts performance. Penalizing intra-domain negatives too harshly destroys fine-grained local structures. The domain-aware reweighting strategy elegantly solves this.
- Gradients as a Proxy for Difficulty: Calculating alignment difficulty across an entire dataset at every step is computationally impossible. Using the running average of mini-batch projector gradient norms provides an efficient, real-time proxy for curriculum learning.
Key Results
UniGraphLM was tested under two primary evaluation settings:
Multi-Domain Multi-Task Learning
When trained jointly on all datasets, UniGraphLM achieved an average performance of 81.95%, outperforming the strongest baseline, TEA-GLM (75.22%), by 6.73 percentage points. It achieved the top score on every single dataset tested.
Cross-Domain and Cross-Task Generalization
To test zero-shot transfer, the model was instruction-tuned only on the Arxiv dataset. When evaluated on unseen domains and tasks, UniGraphLM demonstrated massive improvements over baselines. For instance, on the PubMed dataset, it achieved 77.31% accuracy compared to TEA-GLM's 34.33%. On the edge-level FB15K237 dataset, it reached 55.27% accuracy, while all other baselines struggled below 17%.
Limitations and Future Work
While UniGraphLM excels at standard classification tasks, the authors note that it currently focuses on conventional graph task types (node, edge, and graph classification) over moderate dataset collections. It does not explicitly handle more complex graph reasoning scenarios, such as multi-hop logical queries or generation tasks. The authors point to scaling up the pretraining corpus to larger, more diverse graph-text collections as a vital next step.
Summary updated