Efficient Large Language Models Fine-Tuning on Graphs

Rui Xue, Xipeng Shen, Ruozhou Yu, Xiaorui LiuSubmission to ICLR 2024 · Rejected

TL;DR

Fine-tuning Large Language Models (LLMs) end-to-end alongside Graph Neural Networks (GNNs) on text-attributed graphs usually results in severe computational bottlenecks and memory explosion. This paper introduces LEADING, an efficient end-to-end framework that decouples target and neighbor node encoding and incorporates implicit graph modeling. This approach achieves the memory footprint of standard graph-less LLM fine-tuning while significantly outperforming previous methods in low-data, semi-supervised graph settings.

What Problem Does This Paper Address?

In Text-Attributed Graphs (TAGs)—such as citation networks, social networks, and recommender systems—each node contains rich textual data alongside connectivity patterns. While pre-trained LLMs (e.g., BERT, DeBERTa) can model textual semantics far better than classical bag-of-words or Word2Vec embeddings, training them jointly with GNNs is hindered by:

  1. Inferior Computation Efficiency: The standard recursive neighborhood expansion in GNNs requires repeated, redundant forward passes through giant LLM backbones for neighbor nodes across mini-batches.
  2. Inferior Data Efficiency: In low-resource, semi-supervised scenarios (very few labeled nodes), standard decoupled or pseudo-labeling pipelines (e.g., cascaded models or iterative methods like GLEM) struggle to transfer knowledge effectively, sometimes even performing worse than pre-trained, un-tuned models.

Methods

The authors introduce LEADING to eliminate computation redundancy via two primary strategies:

  • Two-Pipeline Neighbor Decoupling: Instead of passing target and multi-hop neighbor nodes through the full LLM with gradients simultaneously:
  • Pipeline 1 (Target Nodes): Encodes target nodes with gradients and uses these gradients to update the LLM weights.
  • Pipeline 2 (Sampled Nodes): Periodically updates node embeddings for a randomly sampled subset across the whole graph without computing gradients, updating an embedding memory bank to prevent feature staleness.
  • Graph aggregation subsequently pulls neighbor embeddings straight from the memory bank, preventing neighborhood explosion during LLM back-propagation.
  • Implicit Graph Modeling (Propagation Reduction):
  • Rather than storing intermediate states across multi-hop layers (which incurs $O(LNH)$ memory complexity), LEADING employs fixed-point implicit modeling inspired by Neural ODEs and Deep Equilibrium Models (DEQ).
  • Intermediate features are not saved, reducing GNN memory complexity to $O(NH)$ independent of layer count $L$.
  • It reuses feature states and gradients across training iterations to accelerate convergence.
Pipeline 1 (Targets) ---> [LLM + Gradients] ----\
                                                 +--> [Implicit GNN Aggregation] ---> Loss
Pipeline 2 (Neighbors) -> [LLM w/o Gradients] -> Memory Bank -/

Datasets

Experiments evaluate semi-supervised node classification using standard benchmark TAG datasets:

  • Cora: Low-labeling regime (20 labeled nodes per class).
  • PubMed: Low-labeling regime (20 labeled nodes per class).
  • ogbn-arxiv: High-labeling regime (standard 53.7% training split).

No new benchmark datasets were introduced.

Key Insights and Ideas

  • Redundancy is the Bottleneck: On ogbn-arxiv, using a 2-layer GCN with GraphSAGE sampling forces each node to be encoded as a target node once, but as a neighbor node ~19–25 times per epoch on average (jumping to 96 times for a 5-layer model). Decoupling neighbors removes this repeated encoding overhead.
  • Target-Only Gradients Suffice: Subgraph sampling means neighbors rarely have complete neighborhoods within a mini-batch, making their gradients noisy anyway. Restricting backpropagation to target nodes drastically lowers memory without sacrificing downstream quality (less than a 0.5% accuracy difference in ablation experiments).
  • Fixing the Semi-Supervised Gap: Supervised fine-tuning of LLMs on very few labels often causes severe degradation or overfitting. End-to-end graph-aware training acts as a structural regularizer, allowing the model to adapt successfully even with minimal training labels.

Key Results

  • Major Gains in Low-Data Regimes (Cora & PubMed):
  • On Cora, LEADING (with DeBERTa) achieves 80.6% (GCN) and 81.4% (GAT), drastically outperforming standard supervised fine-tuning (59.2% and 57.4%) and matching standard graph models trained from scratch.
  • On PubMed, it achieves 79.5% (GCN) and 79.3% (GAT) vs. ~62% for naive supervised tuning.
  • Competitive in High-Data Regimes (ogbn-arxiv):
  • Achieves 76.1% (GCN) and 77.3% (Rev-GAT), outperforming the iterative state-of-the-art GLEM (75.9% / 76.9%).
  • Scalability and Footprint:
  • Memory overhead is virtually identical to graph-less fine-tuning (~11.7 GB for BERT; ~13.9 GB for DeBERTa), while self-supervised approaches like GIANT crashed with Out-Of-Memory (OOM) errors under the same settings.
  • Wall-clock runtime for DeBERTa on ogbn-arxiv dropped from ~67,634 seconds (GLEM) down to 22,226 seconds for LEADING.

Limitations and Future Work

  • Sequential GPU Pipeline Execution: In the current implementation, the two pipelines execute sequentially on a single GPU, causing a ~0.8× time overhead compared to purely graph-less LLM training (though the authors note this can be mitigated via parallel GPU execution).
  • Model Scale Limitations: The paper benchmarks encoder-only language models (BERT, DeBERTa, DistilBERT). Very large decoder architectures (e.g., LLaMA, Mistral) were not tested.
  • Future Work: The authors propose integrating LEADING with Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, to scale up to massive open-source generative LLMs.

Summary updated

Abstract

Learning from Text-Attributed Graphs (TAGs) has attracted significant attention due to its wide range of real-world applications. The rapid evolution of large language models (LLMs) has revolutionized the way we process textual data, which indicates a strong potential to replace shallow text embedding generally used in Graph Neural Networks (GNNs). However, we find that existing LLM approaches that exploit text information in graphs suffer from inferior computation and data efficiency. In this work, we introduce a novel and efficient approach for the end-to- end fine-tuning of Large Language Models (LLMs) on TAGs, named LEADING. The proposed approach maintains computation cost and memory overhead comparable to the graph-less fine-tuning of LLMs. Moreover, it transfers the rick knowledge in LLMs to downstream graph learning tasks effectively with limited labeled data in semi-supervised learning. Its superior computation and data efficiency are demonstrated through comprehensive experiments, offering a promising solution for a wide range of LLMs and graph learning tasks on TAGs.
BibTeX
@inproceedings{xue2025efficient,
  author = {Rui Xue and Xipeng Shen and Ruozhou Yu and Xiaorui Liu},
  title = {{Efficient Large Language Models Fine-Tuning on Graphs}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2025},
  doi = {10.1145/3711896.3736922},
}
Figure