A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning

Haibo Chen, Xin Wang, Jiaheng Chao, Ling Feng, Wenwu ZhuarXiv 2026 · arXiv

TL;DR

UniGraphLM is a unified graph language model that successfully bridges the gap between diverse graph structures and large language models (LLMs) across multiple domains and tasks. By combining a multi-scale graph neural network (GNN) encoder with a two-stage training strategy—graph-text contrastive pretraining and difficulty-aware curriculum alignment tuning—it achieves state-of-the-art generalization on node, edge, and graph-level tasks.

What Problem Does This Paper Address?

Graph Language Models (GLMs) aim to supercharge LLMs with the structural processing power of Graph Neural Networks (GNNs). Typically, this involves a two-stage pipeline: a GNN encodes the graph, and a projector maps these encodings into tokens the LLM can understand.

However, existing GLMs are heavily siloed. They are usually designed for a single domain (e.g., citation networks) or a single task type (e.g., node classification). When forced to handle diverse graph data, they fail due to massive variations in graph structures, feature distributions, and task granularities. Furthermore, different graph datasets exhibit varying degrees of compatibility with an LLM's token space, making a standard, one-size-fits-all alignment strategy highly suboptimal.

Methods

The authors introduce UniGraphLM, which tackles these challenges using a two-stage framework:

  1. Stage 1: Graph-Text Pair Pretraining
    To learn generalizable, text-aligned representations, the authors use an LLM (Qwen3-8B) to generate natural language descriptions of graph structures and attributes. A multi-scale GNN (GraphSAGE) is then trained alongside a text encoder using a novel Domain-aware Reweighted Contrastive Loss ($L_{DR-CLIP}$). This loss assigns higher weights to negative samples from distant domains, pushing different domains apart in the embedding space while preserving fine-grained, intra-domain semantics. Crucially, task-specific aggregation functions allow the same GNN to output node-, edge-, or graph-level representations.

  2. Stage 2: Curriculum Alignment Tuning
    During instruction tuning, the GNN and LLM are frozen, and only the projector layer is updated. Because different datasets present different alignment difficulties, UniGraphLM introduces an online Difficulty Estimator. It tracks the gradient norm of the projector for each domain during training; a larger gradient norm implies poorer alignment. A scheduler then uses a temperature-scaled softmax to dynamically assign higher training weights to these harder domains.

Figure 1: Overall framework of UniGraphLM. Stage 1: Graph-Text Pair Pretraining. We construct large-scale graph-text pairs across multiple domains and tasks, encode each graph using a multi-scale GNN encoder to produce its task-required node-, edge-, or graph-level representation in a shared space, and train the encoder with a domain-aware reweighted contrastive objective that explicitly accounts for both inter-domain and intra-domain semantic differences. Stage 2: Curriculum Alignment Tuning. During instruction tuning, we estimate domain-level alignment difficulty online from per-domain gradient statistics and adaptively reweight the training objective to focus more on harder domains, leading to more balanced and effective alignment between GNN representations and the LLM.

Datasets

The model was evaluated across a diverse suite of eight benchmark datasets spanning three granularities:

  • Node-level classification: Cora, PubMed, Arxiv (citation networks), and Wiki-CS (web hyperlink network).
  • Edge-level classification: WN18RR and FB15K237 (knowledge graphs).
  • Graph-level classification: ChemPCBA and ChemHIV (molecular datasets).

Key Insights and Ideas

  • Unified Multi-Scale Encoding: Instead of training separate GNNs, UniGraphLM uses a single GNN with task-specific pooling and aggregation layers. By concatenating local node embeddings with global graph context, the model seamlessly handles node, edge, and graph-level tasks simultaneously.
  • Not All Negatives Are Equal: In multi-domain contrastive learning, treating all negative samples uniformly hurts performance. Penalizing intra-domain negatives too harshly destroys fine-grained local structures. The domain-aware reweighting strategy elegantly solves this.
  • Gradients as a Proxy for Difficulty: Calculating alignment difficulty across an entire dataset at every step is computationally impossible. Using the running average of mini-batch projector gradient norms provides an efficient, real-time proxy for curriculum learning.

Key Results

UniGraphLM was tested under two primary evaluation settings:

Multi-Domain Multi-Task Learning

When trained jointly on all datasets, UniGraphLM achieved an average performance of 81.95%, outperforming the strongest baseline, TEA-GLM (75.22%), by 6.73 percentage points. It achieved the top score on every single dataset tested.

Table 1: Performance comparison of different methods on node classification, edge classification, and graph classification tasks under multi-domain and multi-task learning. The highest result is bold, and the second highest result is underlined.

Cross-Domain and Cross-Task Generalization

To test zero-shot transfer, the model was instruction-tuned only on the Arxiv dataset. When evaluated on unseen domains and tasks, UniGraphLM demonstrated massive improvements over baselines. For instance, on the PubMed dataset, it achieved 77.31% accuracy compared to TEA-GLM's 34.33%. On the edge-level FB15K237 dataset, it reached 55.27% accuracy, while all other baselines struggled below 17%.

Table 2: Performance comparison under cross-domain and cross-task settings. The best results are in bold, and the second best are underlined.

Limitations and Future Work

While UniGraphLM excels at standard classification tasks, the authors note that it currently focuses on conventional graph task types (node, edge, and graph classification) over moderate dataset collections. It does not explicitly handle more complex graph reasoning scenarios, such as multi-hop logical queries or generation tasks. The authors point to scaling up the pretraining corpus to larger, more diverse graph-text collections as a vital next step.

Summary updated

Abstract

Leveraging Graph Neural Networks (GNNs) as graph encoders and aligning the resulting representations with Large Language Models (LLMs) through alignment instruction tuning has become a mainstream paradigm for constructing Graph Language Models (GLMs), combining the generalization ability of LLMs with the structural modeling capacity of GNNs. However, existing GLMs that adopt GNNs as graph encoders largely overlook the problem of aligning GNN-encoded representations across domains and tasks with the LLM token space to obtain unified graph tokens, thereby limiting their ability to generalize across diverse graph data. To bridge this gap, we aim to incorporate a multi-domain, multi-task GNN encoder into GLMs and align its representations with LLMs to enable multi-domain, multi-task graph alignment instruction tuning. This alignment problem remains underexplored and poses two key challenges: 1) learning GNN-encoded representations that are simultaneously generalizable across domains and tasks and well aligned with textual semantics is difficult, due to substantial variations in graph structures, feature distributions, and supervision signals, together with the lack of textual-semantic alignment guidance in task-specific GNN training; 2) diverse graph data and task-specific instructions can exhibit different degrees of compatibility with the LLM token space during instruction tuning, leading to varying alignment difficulty and rendering a fixed alignment strategy suboptimal. To tackle these challenges, we propose UniGraphLM, a Unified Graph Language Model that incorporates a multi-domain, multi-task GNN encoder to learn generalizable graph representations aligned with textual semantics, and then adaptively aligns these representations with the LLM.
BibTeX
@misc{chen2026unified,
  author = {Haibo Chen and Xin Wang and Jiaheng Chao and Ling Feng and Wenwu Zhu},
  title = {{A Unified Graph Language Model for Multi-Domain Multi-Task Graph Alignment Instruction Tuning}},
  year = {2026},
  eprint = {2605.12197},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.12197},
}

Loading related content...

Figure