Graph neural architecture search with large language models

Haishuai Wang, Yang Gao, Xin Zheng, Peng Zhang, Jiajun Bu, Philip S. Yu

TL;DR

This paper introduces GNAS-LLM, a novel approach that leverages Large Language Models (LLMs) to automate the design of Graph Neural Network (GNN) architectures. By crafting specific prompts that describe the search space, strategy, and feedback, GNAS-LLM guides LLMs to generate high-performing GNNs, significantly reducing manual effort and outperforming existing GNAS methods.

What Problem Does This Paper Address?

Designing effective Graph Neural Networks (GNNs) for various graph-based tasks is crucial, but the process of finding the optimal architecture is incredibly challenging. Traditional Graph Neural Architecture Search (GNAS) methods, while promising, require extensive human expertise and manual tuning of the search space and strategy. This makes GNAS a labor-intensive and knowledge-heavy endeavor. The paper aims to bridge this gap by using the advanced language understanding and generation capabilities of LLMs to automate and simplify the GNAS process.

Methods

The core innovation is GNAS-LLM, which integrates LLMs into the GNAS framework. Instead of relying on human-defined search spaces and strategies, GNAS-LLM uses specially designed GNAS prompts. These prompts are structured to inform the LLM about:

  • Search Task: The objective (e.g., maximize accuracy on a given dataset).
  • Search Space: This includes defining candidate operations (e.g., GCN, GAT) and, crucially, how they connect. The paper uses an adjacency matrix to represent these connections, which is a key differentiator from simpler operation lists.
  • Search Strategy: Instructions on how to explore the search space, incorporating concepts like exploration (trying new combinations) and exploitation (refining promising architectures), often inspired by reinforcement learning.

The LLM, guided by these prompts, generates candidate GNN architectures. These architectures are then evaluated on the target graph dataset, and the performance feedback is used to refine the GNAS prompts for subsequent iterations. This iterative process allows the LLM to learn and converge towards optimal GNN designs.

Datasets

The paper evaluates GNAS-LLM on several benchmark datasets:

  • NAS-Bench-Graph: This benchmark is used extensively for its pre-generated graph architectures, allowing for direct querying of performance without training. It includes datasets like Cora, Citeseer, Pubmed, and ogbn-arXiv.
  • AutoGEL search space: Used for a case study to demonstrate adaptability to different search spaces.
  • Heterogeneous graph datasets: FB15k-237 and WN18RR are used for link prediction tasks.
  • Large graph dataset: ogbn-products is used to test scalability.

Key Insights and Ideas

  • LLMs as GNAS Controllers: The paper demonstrates that LLMs can effectively act as the "brain" of a GNAS system, capable of understanding complex search spaces and strategies.
  • Adjacency Matrix for Connections: Representing GNN connections using an adjacency matrix in the prompts is crucial for LLMs to grasp the structural relationships between operations, leading to better architecture generation compared to simpler edge lists.
  • Prompt Engineering is Key: The success of GNAS-LLM hinges on well-crafted prompts that clearly define the search task, space, and strategy. This highlights the power of prompt engineering in directing LLM capabilities.
  • Fast Convergence: GNAS-LLM achieves competitive or superior results with significantly fewer search iterations compared to traditional GNAS methods, indicating efficient learning and convergence.
  • Adaptability: The prompt-based approach allows GNAS-LLM to be adaptable to new GNN operations, connections, and even search strategies with minimal modifications.

Key Results

  • Outperforms State-of-the-Art: GNAS-LLM consistently outperforms existing GNAS methods (Random Search, GraphNAS, Genetic-GNN) on benchmark datasets, achieving an average improvement of 0.7% on validation sets and 0.3% on test sets.
  • Significant Improvement on AutoGEL: On the AutoGEL search space, GNAS-LLM shows an average performance enhancement of 0.3% on validation and 1.02% on test sets.
  • Faster Convergence: GNAS-LLM requires only about 15 search iterations to converge, while traditional methods often need 200 iterations to achieve comparable or worse results.
  • Effective on Heterogeneous Graphs: The method demonstrates good generalization by achieving leading results in link prediction tasks on heterogeneous graphs.
  • Scalability: GNAS-LLM shows promise in handling large-scale graph datasets.

Limitations and Future Work

  • Computational Cost of LLMs: The paper acknowledges that LLMs are computationally intensive, which can be a limitation in low-resource settings. Future work could focus on developing more computation-efficient GNAS-LLM variants.
  • Broader GNN Architectures: The authors plan to test the robustness and adaptability of GNAS-LLM with a wider range of GNN architectures, especially in more complex heterogeneous graph scenarios.
  • Information Leakage: While the study suggests that revealing dataset names in prompts doesn't significantly impact performance, further investigation into potential information leakage and its mitigation might be warranted.
  • Search Strategy Sensitivity: The results indicate LLMs are sensitive to the search strategy prompt, suggesting that careful design and exploration of different strategies are important for optimal performance.

Summary updated

Abstract

Graph neural architecture search (GNAS) has shown promising results in finding the best graph neural network architecture on a given graph dataset. However, existing GNAS methods still require intensive human labor and rich domain knowledge when designing the search space and search strategy. To this end, we integrate large language models (LLMs) into GNAS and present a new GNAS model based on LLMs (GNAS-LLM for short). The basic idea of GNAS-LLM is to design a new class of GNAS prompts for LLMs to guide LLMs towards understanding the generative task of graph neural architectures. The prompts consist of descriptions of the search space, search strategy, and search feedback of GNAS. By iteratively running LLMs with the prompts, GNAS-LLM generates more accurate graph neural network architectures with fast convergence. Experimental results show that GNAS-LLM outperforms the state-of-the-art GNAS methods on four benchmark graph datasets, with an average improvement of 0.7% on the validation sets and 0.3% on the test sets. Besides, GNAS-LLM achieves an average improvement of 1.0% on the test sets based on the search space from AutoGEL.
BibTeX
@article{wang2025graph,
  author = {Haishuai Wang and Yang Gao and Xin Zheng and Peng Zhang and Jiajun Bu and Philip S. Yu},
  title = {{Graph neural architecture search with large language models}},
  journal = {Science China Information Sciences},
  year = {2025},
  url = {https://openreview.net/forum?id=0qnQspB7AD},
}
Figure