Abstract
Text-Attributed Graphs (TAGs) integrate structural and textual information, posing challenges for traditional Graph Neural Networks (GNNs) and modern Large Language Models (LLMs). While LLMs offer deep semantic insights, their full-scale fine-tuning is computationally prohibitive and often suffers from domain-specific gaps. We present HiTuner, a hierarchical semantic fusion framework that extracts multi-level hidden states from frozen LLMs and combines them with task-specific knowledge from fine-tuned Pre-trained Language Models (PLMs). By employing a confidence-based fusion mechanism and structural integration via GNNs, HiTuner achieves state-of-the-art performance across six diverse benchmarks, particularly in low-labeling scenarios.
Introduction
Text-Attributed Graphs (TAGs), where nodes represent entities with associated text (e.g., research papers) and edges represent relationships (e.g., citations), are ubiquitous in real-world applications. Traditionally, Graph Neural Networks (GNNs) paired with shallow text encoders have been the standard approach. However, the emergence of Large Language Models (LLMs) like LLaMA provides an opportunity for deeper semantic understanding. Despite their potential, LLMs face two primary hurdles: high resource intensity for fine-tuning and significant domain gaps in specialized fields. HiTuner addresses these challenges by fusing the broad linguistic capabilities of frozen LLMs with the targeted expertise of smaller, fine-tuned PLMs.
Methodology
The HiTuner architecture is designed to bridge the gap between general-purpose LLMs and domain-specific graph tasks through a multi-stage fusion process.
Hierarchical LLM Distillation
Rather than relying solely on the final output layer of an LLM, HiTuner extracts multiple "hidden states" from various layers of a frozen LLM. This captures a hierarchy of linguistic features, from basic syntax to complex semantics. To handle high-dimensional data, the model employs a denoised projector and a threshold-based filter to strip away redundant information.
Task-Oriented Tuners and Fusion
To provide domain-specific "anchoring," a smaller PLM (e.g., BERT) is fine-tuned on the target graph dataset. The representations from both the frozen LLM and the fine-tuned PLM are then processed. A Confidence Network adaptively weights these representations, ensuring that the most relevant features are prioritized for each node based on the model's certainty.
Structural Integration
The fused semantic features are finally integrated with the graph's topology using a GNN (such as GraphSAGE). This ensures that the model leverages both "what is said" (textual attributes) and "who is connected" (graph structure), which the authors found can boost performance by up to 8%.
Experiments and Results
HiTuner was evaluated on six benchmark datasets: CiteSeer, Cora, PubMed (academic), Instagram (social), Amazon Photo (e-commerce), and WikiCS (general knowledge). The experiments focused on low labeling ratios (1% to 10%), reflecting common real-world constraints.
Layer-wise Performance Analysis
A key finding of the study is that the optimal LLM layer for feature extraction varies significantly across different datasets.
As shown in the figure above, deeper layers generally capture richer semantics, but performance can plateau or even degrade beyond a certain depth. This variability justifies HiTuner's hierarchical approach to distillation, as it allows the model to select the most effective representations for a given task.
Comparative Performance
HiTuner achieved an average accuracy of 81.47%, outperforming state-of-the-art methods like TAPE and ENGINE. On the Cora dataset, it reached a peak accuracy of 84.58%. The model demonstrated superior scalability and robustness, particularly in label-scarce environments where the hybrid approach provided a significant advantage over using LLMs or PLMs in isolation.
Conclusion
HiTuner presents an efficient and effective framework for applying LLMs to Text-Attributed Graphs. By combining hierarchical distillation from frozen LLMs with task-specific fine-tuning of smaller models, it overcomes the computational and domain-specific limitations of standard LLM applications. Future work will explore handling graph heterophily—where connected nodes have different labels—and extending this hierarchical fusion approach to multi-modal graphs.
Summary updated