'Hello, World!': Making GNNs Talk with LLMs

Sunwoo Kim, Soo Yong Lee, Jaemin Yoo, Kijung Shin

TL;DR

Graph Lingual Network (GLN) replaces the high-dimensional vector embeddings and matrix aggregations of standard Graph Neural Networks (GNNs) with human-readable text generated by Large Language Models (LLMs). By translating core GNN mechanisms—message passing, graph attention, and initial residual connections—into structured prompts, GLN yields transparent intermediate representations and strong zero-shot performance on graph tasks without any training.

What Problem Does This Paper Address?

Standard GNNs excel at processing graph-structured data by iteratively aggregating neighborhood information across layers. However, their intermediate representations are dense, high-dimensional vectors, making them black boxes: it is difficult to inspect what a node "learns" from its neighbors layer-by-layer.

Meanwhile, prior efforts to pair GNNs with LLMs often require costly fine-tuning, feed raw text summaries into downstream vector GNNs, or prompt LLMs to simply enumerate neighbor lists (which produces cluttered text rather than synthesized representations). The authors ask: Can an LLM act as the message-passing and aggregation engine itself, producing fully interpretable, natural language node representations at each layer?

Methods

GLN reformulates multi-layer GNN operations using pure LLM prompting over text-attributed graphs:

  • Textual Message Passing: At layer $\ell$, the LLM receives the target node’s representation from the previous layer $D^{(\ell-1)}_i$ alongside the representations of its neighbors $\{D^{(\ell-1)}_j : v_j \in \mathcal{N}_i\}$. The LLM synthesizes these into an updated text representation $D^{(\ell)}_i$.
  • Graph Attention Prompting: Instead of computing numerical attention weights, GLN includes explicit prompt instructions directing the LLM to give greater emphasis to neighbors most relevant to the target node's core topic.
  • Initial Residual Connection Prompting: To prevent over-smoothing and contextual drift in higher layers, GLN explicitly itemizes both the raw input attribute ($D^{(0)}_i$) and the previous layer’s output ($D^{(\ell-1)}_i$) within each prompt.
  • Final Multi-Scale Representation: A node’s final embedding is represented as the collection of outputs across all layers $\{D^{(0)}_i, D^{(1)}_i, \dots, D^{(L)}_i\}$, capturing everything from fine-grained local context to high-level multi-hop neighborhoods.
  • Token-Efficient Sampling: GLN samples a fixed subset of neighbors (e.g., 10) and enforces concise output constraints (e.g., 2 paragraphs or 3 sentences).
Layer 0 (Raw Text) ──► [LLM Aggregator + Attention/Residual Prompts] ──► Layer 1 (Local Synthesis)
                             │
                             ▼
                       [LLM Aggregator + Attention/Residual Prompts] ──► Layer 2 (Abstract Context)

Datasets

The authors evaluate GLN on three standard text-attributed graph (TAG) benchmarks:

  1. OGBN-arXiv: A citation network of 169,343 academic papers across 40 arXiv subject classes.
  2. Book-History: An Amazon co-purchase network of 41,551 books across 12 categories.
  3. Wiki-CS: A Wikipedia hyperlink network of 11,701 computer science web pages across 10 categories.

No new datasets were introduced.

Key Insights and Ideas

  • Representations broaden naturally across layers: Qualitative analysis and LLM-as-a-judge evaluations show that Layer 1 representations focus on immediate domain tasks (e.g., attention in NLP), while Layer 2 representations generalize to broader cross-domain applications (e.g., attention in computer vision and graph learning), mirroring how vector GNN receptive fields expand.
  • Attention keeps neighbor summaries targeted: Without the graph attention prompt, LLM summaries devolve into generic overviews of neighbor topics. With attention prompting, the synthesis is actively filtered through the lens of the target node.
  • Residual connections preserve node identity: Explicitly separating the initial attribute from the prior layer’s text prevents the target node from losing its specific identity after multiple hops of aggregation.
  • Human-auditable downstream decisions: Because representations are text, downstream classification errors can be traced directly to specific phrases aggregated from neighboring nodes.

Key Results

The authors evaluated zero-shot node classification (accuracy) and link prediction (Hit-ratio@1) using GPT-4o-mini / GPT-4.1-mini and Claude-3.0-Haiku / Claude-3.5-Haiku:

  • Top Performance Across Baselines: GLN outperformed all zero-shot baselines—including target-only inference (Direct), multi-hop concatenation (All-in-One), and PromptGFM—in 10 out of 12 experimental settings, achieving an average rank of 1.2.
  • Downstream Accuracy: On OGBN-arXiv node classification, GLN achieved 64.0% (GPT) and 67.4% (Claude), improving over Direct (62.3% and 65.8%) and GLN-Base without advanced prompts (63.0% and 67.1%).
  • Link Prediction: On OGBN-arXiv link prediction, GLN reached 93.0% Hit-ratio@1 (GPT) compared to 91.6% for All-in-One.
  • Scalability with Small Language Models (SLMs): Swapping the backbone to a lightweight model (GPT-4.1-nano) reduced per-node encoding time by over 66% (from 8.2s to 2.4s on OGBN-arXiv) while retaining 77.5% classification accuracy.

Limitations and Future Work

  • Theoretical Foundations: Unlike mathematical GNN aggregators, theoretical properties like expressiveness (e.g., Weisfeiler-Lehman bounds) and permutation invariance cannot be easily formalized for an LLM prompt.
  • Computational and Financial Cost: Running LLM calls for multi-layer message passing across thousands of nodes is expensive (costing ~$600 in API calls for the paper’s experiments) and much slower than vectorized matrix multiplications.
  • Graph Structure and Noise: GLN currently only supports text-attributed graphs with pairwise edges (excluding hypergraphs and numerical sensor graphs). Furthermore, synthetic noise experiments showed GLN is sensitive to corrupted node text, requiring explicit text-denoising steps.

Summary updated

Abstract

While graph neural networks (GNNs) have shown remarkable performance across diverse graph-related tasks, their high-dimensional hidden representations render them black boxes. In this work, we propose Graph Lingual Network (GLN), a GNN built on large language models (LLMs), with hidden representations in the form of human-readable text. Through careful prompt design, GLN incorporates not only the message passing module of GNNs but also advanced GNN techniques, including graph attention and initial residual connection. The comprehensibility of GLN’s hidden representations enables an intuitive analysis of how node representations change (1) across layers and (2) under advanced GNN techniques, shedding light on the inner workings of GNNs. Furthermore, we demonstrate that GLN achieves strong zero-shot performance on node classification and link prediction, outperforming existing LLM-based baseline methods.
BibTeX
@inproceedings{kim2025hello,
  author = {Sunwoo Kim and Soo Yong Lee and Jaemin Yoo and Kijung Shin},
  title = {{'Hello, World!': Making GNNs Talk with LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=HDKmkRq2Hp},
}

Loading related content...

Figure