TL;DR
Fine-tuning Large Language Models (LLMs) end-to-end alongside Graph Neural Networks (GNNs) on text-attributed graphs usually results in severe computational bottlenecks and memory explosion. This paper introduces LEADING, an efficient end-to-end framework that decouples target and neighbor node encoding and incorporates implicit graph modeling. This approach achieves the memory footprint of standard graph-less LLM fine-tuning while significantly outperforming previous methods in low-data, semi-supervised graph settings.
What Problem Does This Paper Address?
In Text-Attributed Graphs (TAGs)—such as citation networks, social networks, and recommender systems—each node contains rich textual data alongside connectivity patterns. While pre-trained LLMs (e.g., BERT, DeBERTa) can model textual semantics far better than classical bag-of-words or Word2Vec embeddings, training them jointly with GNNs is hindered by:
- Inferior Computation Efficiency: The standard recursive neighborhood expansion in GNNs requires repeated, redundant forward passes through giant LLM backbones for neighbor nodes across mini-batches.
- Inferior Data Efficiency: In low-resource, semi-supervised scenarios (very few labeled nodes), standard decoupled or pseudo-labeling pipelines (e.g., cascaded models or iterative methods like GLEM) struggle to transfer knowledge effectively, sometimes even performing worse than pre-trained, un-tuned models.
Methods
The authors introduce LEADING to eliminate computation redundancy via two primary strategies:
- Two-Pipeline Neighbor Decoupling: Instead of passing target and multi-hop neighbor nodes through the full LLM with gradients simultaneously:
- Pipeline 1 (Target Nodes): Encodes target nodes with gradients and uses these gradients to update the LLM weights.
- Pipeline 2 (Sampled Nodes): Periodically updates node embeddings for a randomly sampled subset across the whole graph without computing gradients, updating an embedding memory bank to prevent feature staleness.
- Graph aggregation subsequently pulls neighbor embeddings straight from the memory bank, preventing neighborhood explosion during LLM back-propagation.
- Implicit Graph Modeling (Propagation Reduction):
- Rather than storing intermediate states across multi-hop layers (which incurs $O(LNH)$ memory complexity), LEADING employs fixed-point implicit modeling inspired by Neural ODEs and Deep Equilibrium Models (DEQ).
- Intermediate features are not saved, reducing GNN memory complexity to $O(NH)$ independent of layer count $L$.
- It reuses feature states and gradients across training iterations to accelerate convergence.
Pipeline 1 (Targets) ---> [LLM + Gradients] ----\
+--> [Implicit GNN Aggregation] ---> Loss
Pipeline 2 (Neighbors) -> [LLM w/o Gradients] -> Memory Bank -/
Datasets
Experiments evaluate semi-supervised node classification using standard benchmark TAG datasets:
- Cora: Low-labeling regime (20 labeled nodes per class).
- PubMed: Low-labeling regime (20 labeled nodes per class).
- ogbn-arxiv: High-labeling regime (standard 53.7% training split).
No new benchmark datasets were introduced.
Key Insights and Ideas
- Redundancy is the Bottleneck: On
ogbn-arxiv, using a 2-layer GCN with GraphSAGE sampling forces each node to be encoded as a target node once, but as a neighbor node ~19–25 times per epoch on average (jumping to 96 times for a 5-layer model). Decoupling neighbors removes this repeated encoding overhead. - Target-Only Gradients Suffice: Subgraph sampling means neighbors rarely have complete neighborhoods within a mini-batch, making their gradients noisy anyway. Restricting backpropagation to target nodes drastically lowers memory without sacrificing downstream quality (less than a 0.5% accuracy difference in ablation experiments).
- Fixing the Semi-Supervised Gap: Supervised fine-tuning of LLMs on very few labels often causes severe degradation or overfitting. End-to-end graph-aware training acts as a structural regularizer, allowing the model to adapt successfully even with minimal training labels.
Key Results
- Major Gains in Low-Data Regimes (Cora & PubMed):
- On Cora, LEADING (with DeBERTa) achieves 80.6% (GCN) and 81.4% (GAT), drastically outperforming standard supervised fine-tuning (59.2% and 57.4%) and matching standard graph models trained from scratch.
- On PubMed, it achieves 79.5% (GCN) and 79.3% (GAT) vs. ~62% for naive supervised tuning.
- Competitive in High-Data Regimes (ogbn-arxiv):
- Achieves 76.1% (GCN) and 77.3% (Rev-GAT), outperforming the iterative state-of-the-art GLEM (75.9% / 76.9%).
- Scalability and Footprint:
- Memory overhead is virtually identical to graph-less fine-tuning (~11.7 GB for BERT; ~13.9 GB for DeBERTa), while self-supervised approaches like GIANT crashed with Out-Of-Memory (OOM) errors under the same settings.
- Wall-clock runtime for DeBERTa on
ogbn-arxivdropped from ~67,634 seconds (GLEM) down to 22,226 seconds for LEADING.
Limitations and Future Work
- Sequential GPU Pipeline Execution: In the current implementation, the two pipelines execute sequentially on a single GPU, causing a ~0.8× time overhead compared to purely graph-less LLM training (though the authors note this can be mitigated via parallel GPU execution).
- Model Scale Limitations: The paper benchmarks encoder-only language models (BERT, DeBERTa, DistilBERT). Very large decoder architectures (e.g., LLaMA, Mistral) were not tested.
- Future Work: The authors propose integrating LEADING with Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA, to scale up to massive open-source generative LLMs.
Summary updated