Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation

Yuxiang Wang, Xinnan Dai, Wenqi Fan, Yao Ma

Abstract

Graph-structured data has become increasingly prevalent across various domains, raising the demand for effective models to handle graph tasks like node classification and link prediction. Traditional graph learning models like Graph Neural Networks (GNNs) have made significant strides, but their capabilities in handling graph data remain limited in certain contexts. In recent years, large language models (LLMs) have emerged as promising candidates for graph tasks, yet most studies focus primarily on performance benchmarks and fail to address their broader potential, including their ability to handle limited data, their transferability across tasks, and their robustness. In this work, we provide a comprehensive exploration of LLMs applied to graph tasks. We evaluate the performance of pure LLMs, including those without parameter optimization and those fine-tuned with instructions, across various scenarios. Our analysis goes beyond accuracy, assessing LLM ability to perform in few-shot/zero-shot settings, transfer across domains, understand graph structures, and demonstrate robustness in challenging scenarios. We conduct extensive experiments with 16 graph learning models alongside 6 LLMs (e.g., Llama3B, GPT-4o, Qwen-plus), comparing their performance on datasets like Cora, PubMed, ArXiv, and Products. Our findings show that LLMs, particularly those with instruction tuning, outperform traditional models in few-shot settings, exhibit strong domain transferability, and demonstrate excellent generalization and robustness. This work offers valuable insights into the capabilities of LLMs for graph learning, highlighting their advantages and potential for real-world applications, and paving the way for future research in this area.

Introduction

Graphs—data structures composed of nodes and edges—are ubiquitous, appearing in social networks, citation indices, and molecular structures. Traditionally, Graph Neural Networks (GNNs) have been the standard for analyzing such data. However, GNNs often require large amounts of labeled data and struggle with cross-domain generalization. Large Language Models (LLMs) present a potential alternative due to their inherent reasoning and few-shot learning capabilities. This paper investigates the performance of "pure" LLMs—models that process graph data solely through text descriptions—on fundamental graph tasks like node classification and link prediction.

Methodology

The study evaluates two primary approaches for utilizing LLMs in graph learning: off-the-shelf usage and instruction tuning.

Graph-to-Text Encoding

To enable LLMs to process graph structures, the authors developed a standardized encoding scheme. This includes "Ego" (target node attributes only), "1-hop", and "2-hop" neighborhoods, where the attributes of immediate and secondary neighbors are included in the prompt. Explicit structural language, such as "neighbor" and "hop," is used to guide the model's reasoning and switch it into a graph-processing mode.

Instruction Tuning and Training

The researchers fine-tuned open-source models (Llama-3B and 8B) using Low-Rank Adaptation (LoRA). They also explored Continuous Pre-training (CPT), where models are first trained on unsupervised link prediction tasks before being tuned for specific downstream applications. This process aims to instill a "graph inductive bias" similar to the message-passing mechanism in GNNs.

The overall experimental pipeline, illustrating graph encoding into text and the instruction tuning process for LLMs.

Experimental Setup

The benchmark utilizes a diverse set of Text-Attributed Graphs (TAGs), including citation networks (Cora, PubMed, ArXiv), e-commerce data (Products), and social networks (Reddit). To address potential data leakage, a custom ArXiv-2023 dataset was created, containing papers published after the training cutoff of the evaluated LLMs, ensuring that the models were not simply recalling information from their pre-training phase.

Key Results and Insights

The investigation yielded several critical findings regarding the efficacy of pure LLMs:

  • Superiority of Instruction Tuning: Small, instruction-tuned models like Llama-3B frequently outperformed much larger, untuned models like GPT-4o. Tuning effectively teaches the model to simulate graph logic.
  • Few-Shot Excellence: Unlike GNNs, which struggle with limited labels, tuned LLMs demonstrated exceptional performance in few-shot settings (e.g., 5 or 10 examples per class), showing they can generalize from very little data.
  • Domain Transferability: Models tuned on one domain (e.g., ArXiv citation data) showed remarkable performance when transferred to entirely different domains (e.g., Products e-commerce data). This suggests the LLM learns a general "graph logic" that transcends specific topics.
Comparison of domain transferability across different datasets and model scales, highlighting the effectiveness of instruction-tuned LLMs compared to specialized assistants.
  • Robustness: Tuned LLMs proved more resilient than traditional GNNs to structural noise, such as the random removal of edges, maintaining higher accuracy in degraded data scenarios.
  • State-of-the-Art Performance: The tuned Llama-8B model achieved an average accuracy of 86.35% across node classification tasks, setting a new benchmark for pure LLM approaches and surpassing many hybrid models.

Discussion and Future Work

While pure LLMs show great promise, challenges remain. The text-based representation of large neighborhoods can lead to long prompts that exceed context windows in dense graphs. Additionally, the computational cost of training LLMs for graphs is significantly higher (roughly 100x) than for GNNs. Future research should focus on more efficient graph-to-text representations and extending these capabilities to non-textual graphs, such as purely mathematical or chemical structures.

Summary updated

Abstract

Graph-structured data has become increasingly prevalent across various domains, raising the demand for effective models to handle graph tasks like node classification and link prediction. Traditional graph learning models like Graph Neural Networks (GNNs) have made significant strides, but their capabilities in handling graph data remain limited in certain contexts. In recent years, large language models (LLMs) have emerged as promising candidates for graph tasks, yet most studies focus primarily on performance benchmarks and fail to address their broader potential, including their ability to handle limited data, their transferability across tasks, and their robustness. In this work, we provide a comprehensive exploration of LLMs applied to graph tasks. We evaluate the performance of pure LLMs, including those without parameter optimization and those fine-tuned with instructions, across various scenarios. Our analysis goes beyond accuracy, assessing LLM ability to perform in few-shot/zero-shot settings, transfer across domains, understand graph structures, and demonstrate robustness in challenging scenarios. We conduct extensive experiments with 16 graph learning models alongside 6 LLMs (e.g., Llama3B, GPT-4o, Qwen-plus), comparing their performance on datasets like Cora, PubMed, ArXiv, and Products. Our findings show that LLMs, particularly those with instruction tuning, outperform traditional models in few-shot settings, exhibit strong domain transferability, and demonstrate excellent generalization and robustness. This work offers valuable insights into the capabilities of LLMs for graph learning, highlighting their advantages and potential for real-world applications, and paving the way for future research in this area. Codes and datasets are released in https://github.com/myflashbarry/LLM-benchmarking.
BibTeX
@misc{wang2025exploring,
  author = {Yuxiang Wang and Xinnan Dai and Wenqi Fan and Yao Ma},
  title = {{Exploring Graph Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation}},
  year = {2025},
  eprint = {2502.18771},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.18771},
}
Figure