Distilling Large Language Models for Text-Attributed Graph Learning

Bo Pan, Zheng Zhang, Yifei Zhang, Yuntong Hu, Liang Zhao

Introduction

Text-Attributed Graphs (TAGs) represent a fundamental data structure in modern information systems, where nodes represent text-heavy entities—such as research papers, products, or social media profiles—and edges represent their relationships. Traditionally, Graph Neural Networks (GNNs) have been the standard for learning from TAGs. However, GNNs typically require extensive human-annotated data to achieve high performance.

Recently, Large Language Models (LLMs) have demonstrated remarkable zero-shot reasoning capabilities on graph-related tasks. Despite their power, LLMs are often impractical for production-scale TAG applications due to three primary constraints: the high cost of API calls, privacy concerns regarding sensitive data, and the architectural mismatch between generative text models and discriminative graph models. This paper introduces a novel framework designed to bridge this gap by distilling the reasoning power of LLMs into localized, efficient GNNs.

The Challenge: From Eloquence to Efficiency

The core research question is how to transfer the complex reasoning of an LLM into a succinct local model without sacrificing performance or incurring the overhead of the LLM at inference time. Simple label distillation—where a student model merely mimics the LLM's final output—often fails to capture the underlying logic that makes LLMs effective. To address this, the authors propose a method that focuses on "rationales" rather than just labels.

Methodology

The proposed framework utilizes a two-stage distillation process centered around an Interpreter Model that acts as a bridge between the LLM (teacher) and the GNN (student).

1. Rationale Generation

Instead of treating the LLM as a simple annotator, the framework prompts the LLM to generate textual rationales. These rationales identify three key components:

  • Keywords: The most significant words within a node's text.
  • Key Links: The specific neighboring nodes that influence a classification.
  • Key Messages: The relevant information passed from those neighbors.

2. The Interpreter and Student Alignment

The Interpreter Model is trained using these rationales as augmented features, essentially learning the task with a "cheatsheet" provided by the LLM. The Student Model, a standard GNN, is then trained to align its latent embeddings with those of the Interpreter. This alignment process—the framework's "secret sauce"—minimizes the discrepancy between the raw input processed by the student and the rationale-enhanced input processed by the interpreter. Consequently, the student learns to implicitly mimic the LLM's reasoning process using only the original graph data.

Experimental Evaluation

The framework was evaluated on four major TAG benchmarks, including classic citation networks (Cora, Pubmed) and the massive Amazon co-purchase network (ogbn-products). Notably, the authors introduced the arxiv-2023 dataset, which consists of papers published after the knowledge cutoff of models like GPT-3.5. This ensures that the results reflect the model's reasoning ability rather than simple data leakage from the LLM's training set.

Key Results

  • Superior Accuracy: The method outperformed the strongest baselines by an average of 6.2% across all datasets. On the arxiv-2023 dataset, it achieved a 10.3% improvement, demonstrating its robustness against data leakage.
  • Data Efficiency: In low-data scenarios (using only 0.1% of training labels), the distillation method maintained a significant lead, proving that LLM rationales are highly effective teachers when human labels are scarce.
  • Inference Efficiency: Once trained, the student GNN operates independently of the LLM. This eliminates the need for expensive API calls and ensures that sensitive data remains local, solving the scalability and privacy issues inherent in LLM-based inference.

Insights and Discussion

A critical finding of the study is that "Rationales > Labels." Distilling the reasoning process provides a much stronger learning signal than distilling final predictions. Furthermore, ablation studies revealed that structural rationales (Key Links and Key Messages) contributed more to performance gains than textual rationales (Keywords). This suggests that LLMs can significantly enhance a GNN's understanding of graph topology.

However, the framework does involve trade-offs. While inference is cheap, the training phase is more token-intensive than simple annotation (requiring approximately 4.2M tokens compared to 1.4M in the authors' tests). Additionally, the student's performance remains bounded by the quality of the LLM's rationales; hallucinations in the teacher model can lead to suboptimal features in the interpreter.

Conclusion

This research provides a scalable pathway for integrating the advanced reasoning of LLMs into the specialized architecture of GNNs. By focusing on rationale distillation and embedding alignment, the framework enables high-performance graph learning that is both cost-effective and privacy-preserving. Future work may extend this approach beyond node classification to tasks such as link prediction and graph-level reasoning.

Summary updated

BibTeX
@inproceedings{pan2024distilling,
  author = {Bo Pan and Zheng Zhang and Yifei Zhang and Yuntong Hu and Liang Zhao},
  title = {{Distilling Large Language Models for Text-Attributed Graph Learning}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2024},
  doi = {10.1145/3627673.3679830},
}
Figure