TL;DR
LEADING is a novel framework that enables efficient, end-to-end fine-tuning of Language Models (LMs) on Text-Attributed Graphs (TAGs). By decoupling neighbor encoding and employing implicit graph modeling, it achieves state-of-the-art performance on major benchmarks while keeping memory costs as low as standard, graph-less LM fine-tuning.
What Problem Does This Paper Address?
In the real world, data often comes as Text-Attributed Graphs (TAGs)—think of citation networks where nodes are papers (text) and edges are citations, or social networks where nodes are users (profiles/posts) and edges are friendships.
While Graph Neural Networks (GNNs) are great at handling structure and LMs are masters of text, combining them is notoriously difficult. The primary culprit is the "neighbor explosion" problem: to compute the representation of a single node, a GNN needs to aggregate information from its neighbors, their neighbors, and so on. If every one of those neighbors requires a pass through a massive LM (like BERT or DeBERTa), the computational and memory costs skyrocket.
Existing solutions usually take a "cascaded" approach (fine-tune the LM first, then use its static outputs for the GNN), which is efficient but misses out on the rich, joint signals available in end-to-end training. Iterative methods exist but are often too slow or memory-intensive for large-scale graphs.
Methods
The authors introduce LEADING (Language models fine-tuning on Graphs), which tackles computation and propagation redundancy through two main techniques:
- Neighbor Decoupling: The authors observed a massive "encoding redundancy" in standard mini-batch GNN training. As shown in Figure 1, a single node might be encoded dozens of times as a neighbor for different target nodes in a single epoch. Encoding Redundancy in Mini-batch GNNs LEADING splits the process into two pipelines (Figure 2). Pipeline 1 encodes only the target nodes with gradients for fine-tuning. Pipeline 2 encodes a random batch of nodes without gradients and caches them in a memory bank. When the GNN needs neighbor features, it simply pulls them from this cache, drastically reducing LM passes. LEADING: two-pipeline training process
- Implicit Graph Modeling: To capture long-range dependencies without the memory overhead of deep GNN layers, LEADING uses a fixed-point solver approach (based on APPNP). Instead of storing intermediate activations for every layer, it treats the GNN's final state as an equilibrium point. This makes the memory cost independent of the number of GNN layers, allowing for "deep" structural reasoning at a "shallow" memory price.
Datasets
The framework was evaluated on four standard graph benchmarks:
- Cora & PubMed: Small-scale citation graphs used to test performance in low-labeling (semi-supervised) regimes.
- ogbn-arxiv: A large-scale citation network from the Open Graph Benchmark.
- ogbn-products: A massive Amazon product co-purchasing network.
Key Insights and Ideas
- Redundancy is the Real Bottleneck: Statistical analysis on
ogbn-arxivshowed that in a 2-layer GNN, a node is encoded as a neighbor ~19 times on average per epoch, but only once as a target. Decoupling these roles is the "aha moment" that makes end-to-end training feasible. - Staleness is a Feature, Not a Bug: Because LM embeddings don't change drastically between iterations during fine-tuning, using cached (slightly "stale") embeddings for neighbors provides sufficient signal for the GNN without sacrificing accuracy.
- End-to-End is Critical for Small Data: The paper demonstrates that in low-label settings (like Cora), cascaded methods fail to transfer LM knowledge effectively. LEADING fills this gap, showing that joint training is essential when labels are scarce.
- Scalability Breakthrough: Unlike standard GNN+LM setups that crash due to Out-of-Memory (OOM) errors on large graphs, LEADING's memory footprint remains flat regardless of how many "hops" (neighbor layers) are added.
Key Results
- SOTA Performance: LEADING achieved state-of-the-art results on the
ogbn-arxivleaderboard. When combined with TAPE features, it reached an accuracy of 78.2%. - Massive Memory Savings: On
ogbn-arxiv, standard LM+GNN approaches OOM'd, while LEADING maintained a memory cost of ~26GB—comparable to fine-tuning an LM on text alone. - Data Efficiency: On the Cora dataset with low labeling, LEADING boosted DeBERTa's accuracy from 59.2% (standard fine-tuning) to 80.6%.
- Encoding Efficiency: As shown in Figure 5, LEADING ensures each node is encoded only twice per epoch, whereas methods like GraphSAGE or GAS can encode nodes up to 100 times depending on the batch size and hop count. Encoding Times Comparison
Limitations and Future Work
The authors note that their two-pipeline approach currently runs sequentially on a single GPU, which makes the wall-clock running time about 0.8x higher than simple supervised fine-tuning. However, they point out that these pipelines are inherently independent and could be easily accelerated through parallel computing across multiple GPUs.
Future work could involve extending this framework to even larger models (LLMs) or exploring its application in more diverse graph tasks like heterogeneous graph learning.
Summary updated