STAGE: Simplified Text-Attributed Graph Embeddings using Pre-trained LLMs

Aaron Zolnai-Lucas, Jack Boylan, Chris Hokamp, Parsa GhaffariACL 2024

TL;DR

This paper introduces STAGE, a novel method for creating node embeddings in Text-Attributed Graphs (TAGs) by leveraging pre-trained Large Language Models (LLMs) without requiring LLM finetuning. STAGE offers a simpler, more efficient approach to node classification on TAGs, achieving competitive results while significantly reducing computational costs and complexity compared to existing state-of-the-art methods.

What Problem Does This Paper Address?

Many real-world datasets can be represented as Text-Attributed Graphs (TAGs), where nodes have associated textual information. Effectively learning from these graphs requires integrating both the graph structure and the textual content. Existing methods often involve complex, multi-stage processes, including iterative training of LLMs and Graph Neural Networks (GNNs) or cascading approaches that require extensive finetuning of LLMs and potentially multiple training stages for different components. This complexity makes these methods computationally expensive and difficult to apply to large, real-world graphs. The paper aims to simplify this process by developing a straightforward yet effective method for generating node embeddings from textual attributes using pre-trained LLMs.

Methods

The core of STAGE is a two-step cascading approach:

  1. Zero-shot LLM-based Embedding Generation: An off-the-shelf, pre-trained LLM is used to encode the textual attributes (title and abstract) of each node. Crucially, this step does not involve finetuning the LLM or using prompt-based data augmentation. The LLM is used directly as a text embedding model.
  2. GNN Ensemble Training: The generated text embeddings are then used as node features for training an ensemble of GNN architectures (including GCN, SAGE, RevGAT, and MLP). The predictions from these individual GNNs are mean-pooled to produce the final node predictions.

The paper also explores diffusion-pattern GNNs (Simple-GCN, SIGN) to enhance scalability for larger graphs.

Datasets

The proposed STAGE method was evaluated on five benchmark TAG datasets:

  • Cora: A citation network of scientific publications.
  • PubMed: A citation network of diabetes-related scientific publications.
  • ogbn-arxiv: A dataset of arXiv papers linked by citations.
  • ogbn-products: An Amazon product co-purchasing network.
  • tape-arxiv23: A dataset focusing on arXiv papers published after GPT-3.5's knowledge cut-off.

Key Insights and Ideas

  • LLMs as Zero-Shot Embedders are Sufficient: The paper demonstrates that pre-trained LLMs, when used directly without finetuning or complex prompting, can generate powerful node embeddings for TAGs. This significantly simplifies the pipeline.
  • Ensembling GNNs Boosts Robustness: Combining predictions from multiple GNN architectures consistently improved performance and robustness across different datasets and embedding models.
  • Instruction-biased Embeddings Have Minimal Impact: Contrary to common practice, the study found that varying task-specific instructions for the LLM had little to no significant effect on downstream node classification performance in their experiments.
  • Simplicity vs. Performance Trade-off: STAGE achieves competitive results with significantly less complexity and computational cost than methods like TAPE, which rely on extensive finetuning and LLM API calls.

Key Results

  • STAGE achieved competitive node classification accuracy across all tested benchmarks, often lagging behind state-of-the-art methods by only a small margin (e.g., ~5% on Cora, ~3.5% on PubMed compared to TAPE).
  • The training time for STAGE was dramatically reduced. For instance, training on ogbn-arxiv using an ensemble of GNNs and an MLP head could be performed on a single consumer-grade GPU in under 5 minutes, whereas methods like TAPE took significantly longer (e.g., 192 minutes for TAPE on ogbn-arxiv).
  • Ensembling GNN models consistently ranked among the top performers, delivering an average performance increase of 1%.
  • RevGAT was identified as a consistently strong individual GNN architecture.
  • Diffusion-pattern GNNs like SIGN showed promise for scalability, with SIGN emerging as a strong performer on the Cora dataset.

Limitations and Future Work

  • Scalability of LLM API Calls: While STAGE avoids LLM finetuning, other methods like TAPE rely on LLM API calls for data augmentation. This can be a bottleneck for large-scale datasets due to cost and rate-limiting.
  • Instruction Bias: The paper found minimal impact from instruction biasing, which contrasts with some prior work. This suggests further investigation into effective prompting strategies might be beneficial.
  • Future Work: The authors suggest exploring more advanced LLM-GNN integration techniques, such as Expectation-Maximization or joint model configurations. They also highlight the challenge of large, variable batch sizes during LLM finetuning and aim to examine the scalability of diffusion-pattern GNNs on larger datasets.

Summary updated

BibTeX
@inproceedings{zolnailucas2024stage,
  author = {Aaron Zolnai-Lucas and Jack Boylan and Chris Hokamp and Parsa Ghaffari},
  title = {{STAGE: Simplified Text-Attributed Graph Embeddings using Pre-trained LLMs}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure