Abstract
While graph neural networks (GNNs) have shown remarkable performance across diverse graph-related tasks, their high-dimensional hidden representations render them black boxes. In this work, we propose Graph Lingual Network (GLN), a GNN built on large language models (LLMs), with hidden representations in the form of human-readable text. Through careful prompt design, GLN incorporates not only the message passing module of GNNs but also advanced GNN techniques, including graph attention and initial residual connection. The comprehensibility of GLN's hidden representations enables an intuitive analysis of how node representations change (1) across layers and (2) under advanced GNN techniques, shedding light on the inner workings of GNNs. Furthermore, we demonstrate that GLN achieves strong zero-shot performance on node classification and link prediction, outperforming existing LLM-based baseline methods.
Introduction
Graph Neural Networks (GNNs) are the gold standard for processing relational data like social networks and citation graphs. However, they suffer from a significant transparency issue: they represent nodes as high-dimensional vector embeddings. To a human, these vectors are meaningless, making it nearly impossible to understand the reasoning behind a specific prediction.
We introduce the Graph Lingual Network (GLN), a framework that replaces these "black box" embeddings with human-readable text generated by Large Language Models (LLMs). By leveraging the natural language reasoning of models like GPT-4 and Claude, GLN performs core GNN operations—message passing and aggregation—entirely in text, making the model interpretable by design.
Methodology
GLN mimics the architecture of a standard GNN but operates in the natural language domain. The "hidden state" of a node is a text summary that evolves through layers of refinement.
Message Passing via Prompting
For a target node, the LLM is provided with the node's current description and the descriptions of its neighbors. It is then prompted to "refine" the target node's description by incorporating relevant information from the neighborhood. To manage computational load, we utilize neighbor sampling and constrain output lengths.
Advanced GNN Techniques
We translate mathematical GNN operations into natural language instructions to enhance performance:
- Graph Attention: The prompt instructs the LLM to "give more emphasis to those [neighbors] more relevant to the target."
- Initial Residual Connection: To prevent "oversmoothing" (where a node's unique identity is lost in deep layers), the prompt explicitly includes the node's raw, initial text attributes at every layer.
Experimental Results
We evaluated GLN on three Text-Attributed Graph (TAG) benchmarks: OGBN-arXiv (citations), Book-History (co-purchases), and Wiki-CS (Wikipedia hyperlinks).
Zero-Shot Performance
GLN demonstrated significant improvements over existing LLM-based baselines, such as direct prompting or simple neighborhood summarization. As shown in the performance table, GLN achieved an Average Ranking (A.R.) of 1.2, outperforming the next best baseline (A.R. 2.2). On the OGBN-arXiv dataset, GLN reached a node classification accuracy of 67.4% using Claude 3.
Qualitative Insights
The human-readable hidden states allow for direct observation of GNN dynamics:
- Layer-wise Generality: Layer 1 representations focus on local, fine-grained features, while Layer 2 outputs naturally become more abstract and capture broader community context.
- Semantic Anchoring: Initial residual prompts act as a semantic anchor, ensuring the node's core identity is preserved even as it incorporates neighborhood information.
Discussion and Limitations
While GLN offers unprecedented transparency, the computational cost of LLM APIs remains a barrier to scaling for massive graphs. Additionally, while textual attention and residuals work empirically, proving mathematical properties like permutation invariance for LLM-based aggregators remains a theoretical challenge. Future work will focus on integrating Small Language Models (SLMs) and extending the framework to non-textual graph data.
Conclusion
GLN bridges the gap between the relational power of GNNs and the interpretability of natural language. By making the hidden states of a graph network readable, we provide a new tool for understanding and deploying explainable AI in complex relational domains.
Summary updated