Efficient End-to-end Language Model Fine-tuning on Graphs

Rui Xue, Xipeng Shen, Ruozhou Yu, Xiaorui Liu

TL;DR

LEADING is a novel framework that enables efficient, end-to-end fine-tuning of Language Models (LMs) on Text-Attributed Graphs (TAGs). By decoupling neighbor encoding and employing implicit graph modeling, it achieves state-of-the-art performance on major benchmarks while keeping memory costs as low as standard, graph-less LM fine-tuning.

What Problem Does This Paper Address?

In the real world, data often comes as Text-Attributed Graphs (TAGs)—think of citation networks where nodes are papers (text) and edges are citations, or social networks where nodes are users (profiles/posts) and edges are friendships.

While Graph Neural Networks (GNNs) are great at handling structure and LMs are masters of text, combining them is notoriously difficult. The primary culprit is the "neighbor explosion" problem: to compute the representation of a single node, a GNN needs to aggregate information from its neighbors, their neighbors, and so on. If every one of those neighbors requires a pass through a massive LM (like BERT or DeBERTa), the computational and memory costs skyrocket.

Existing solutions usually take a "cascaded" approach (fine-tune the LM first, then use its static outputs for the GNN), which is efficient but misses out on the rich, joint signals available in end-to-end training. Iterative methods exist but are often too slow or memory-intensive for large-scale graphs.

Methods

The authors introduce LEADING (Language models fine-tuning on Graphs), which tackles computation and propagation redundancy through two main techniques:

  1. Neighbor Decoupling: The authors observed a massive "encoding redundancy" in standard mini-batch GNN training. As shown in Figure 1, a single node might be encoded dozens of times as a neighbor for different target nodes in a single epoch. Encoding Redundancy in Mini-batch GNNs LEADING splits the process into two pipelines (Figure 2). Pipeline 1 encodes only the target nodes with gradients for fine-tuning. Pipeline 2 encodes a random batch of nodes without gradients and caches them in a memory bank. When the GNN needs neighbor features, it simply pulls them from this cache, drastically reducing LM passes. LEADING: two-pipeline training process
Figure 1: Encoding Redundancy in Mini-batch GNNs.
Figure 2: LEADING: two-pipeline training process. (a) a ran- domly sampled batch is encoded in pipeline 2 and is stored in memory. (b) only the target nodes within the neighbor- sampled batch are encoded with gradients in pipeline 1. (c) Neighbor nodes’ embeddings are retrieved and the resulting subgraph is then fed into GNNs. (d) Gradients from target nodes are employed to fine-tune the language models.
  1. Implicit Graph Modeling: To capture long-range dependencies without the memory overhead of deep GNN layers, LEADING uses a fixed-point solver approach (based on APPNP). Instead of storing intermediate activations for every layer, it treats the GNN's final state as an equilibrium point. This makes the memory cost independent of the number of GNN layers, allowing for "deep" structural reasoning at a "shallow" memory price.

Datasets

The framework was evaluated on four standard graph benchmarks:

  • Cora & PubMed: Small-scale citation graphs used to test performance in low-labeling (semi-supervised) regimes.
  • ogbn-arxiv: A large-scale citation network from the Open Graph Benchmark.
  • ogbn-products: A massive Amazon product co-purchasing network.

Key Insights and Ideas

  • Redundancy is the Real Bottleneck: Statistical analysis on ogbn-arxiv showed that in a 2-layer GNN, a node is encoded as a neighbor ~19 times on average per epoch, but only once as a target. Decoupling these roles is the "aha moment" that makes end-to-end training feasible.
  • Staleness is a Feature, Not a Bug: Because LM embeddings don't change drastically between iterations during fine-tuning, using cached (slightly "stale") embeddings for neighbors provides sufficient signal for the GNN without sacrificing accuracy.
  • End-to-End is Critical for Small Data: The paper demonstrates that in low-label settings (like Cora), cascaded methods fail to transfer LM knowledge effectively. LEADING fills this gap, showing that joint training is essential when labels are scarce.
  • Scalability Breakthrough: Unlike standard GNN+LM setups that crash due to Out-of-Memory (OOM) errors on large graphs, LEADING's memory footprint remains flat regardless of how many "hops" (neighbor layers) are added.

Key Results

  • SOTA Performance: LEADING achieved state-of-the-art results on the ogbn-arxiv leaderboard. When combined with TAPE features, it reached an accuracy of 78.2%.
  • Massive Memory Savings: On ogbn-arxiv, standard LM+GNN approaches OOM'd, while LEADING maintained a memory cost of ~26GB—comparable to fine-tuning an LM on text alone.
  • Data Efficiency: On the Cora dataset with low labeling, LEADING boosted DeBERTa's accuracy from 59.2% (standard fine-tuning) to 80.6%.
  • Encoding Efficiency: As shown in Figure 5, LEADING ensures each node is encoded only twice per epoch, whereas methods like GraphSAGE or GAS can encode nodes up to 100 times depending on the batch size and hop count. Encoding Times Comparison
Figure 5: Encoding Times

Limitations and Future Work

The authors note that their two-pipeline approach currently runs sequentially on a single GPU, which makes the wall-clock running time about 0.8x higher than simple supervised fine-tuning. However, they point out that these pipelines are inherently independent and could be easily accelerated through parallel computing across multiple GPUs.

Future work could involve extending this framework to even larger models (LLMs) or exploring its application in more diverse graph tasks like heterogeneous graph learning.

Summary updated

Abstract

Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications. The rapid evolution of language models (LMs) has revolutionized the way we process textual data, which indicates a strong potential to replace shallow text embedding generally used in Graph Neural Networks (GNNs). However, we find that existing LM approaches that exploit text information in graphs suffer from inferior computation and data efficiency. In this study, we introduce LEADING, a novel and efficient approach for end-to-end fine-tuning of language models on TAGs. To enhance data efficiency, LEADING efficiently transfers rich knowledge from LMs to downstream graph learning tasks with limited labeled data by employing end-to-end training of LMs and GNNs in a semi-supervised learning setting. To address associated computation efficiency issues, it introduces two techniques: neighbor decoupling targeting LMs and implicit graph modeling targeting GNNs, respectively. Our proposed approach demonstrates superior performance, achieving state-of-the-art (SOTA) results on the ogbn-arxiv leaderboard, while maintaining computation cost and memory overhead comparable to graph-less fine-tuning of LMs. Through comprehensive experiments, we showcase its superior computation and data efficiency, presenting a promising solution for various LMs and graph learning tasks on TAGs.
BibTeX
@inproceedings{xue2025efficient,
  author = {Rui Xue and Xipeng Shen and Ruozhou Yu and Xiaorui Liu},
  title = {{Efficient End-to-end Language Model Fine-tuning on Graphs}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
  doi = {10.1145/3711896.3736922},
}
Figure