GNN2GNN: Graph Neural Networks to Generate Neural Networks

Andrea Agiollo, Andrea OmiciniUAI 2022 · OpenReview

Abstract

The success of Neural Networks (NNs) is tightly linked with their architectural design—a complex problem by itself. We here introduce a novel framework leveraging Graph Neural Networks to Generate Neural Networks (GNN2GNN) where powerful NN architectures can be learned out of a set of available architecture-performance pairs. GNN2GNN relies on a three-way adversarial training of GNN, to optimise a generator model capable of producing predictions about powerful NN architectures. Unlike Neural Architecture Search (NAS) techniques proposing efficient searching algorithms over a set of NN architectures, GNN2GNN relies on learning NN architectural design criteria. GNN2GNN learns to propose NN architectures in a single step—i.e., training of the generator—, overcoming the recursive approach characterising NAS. Therefore, GNN2GNN avoids the expensive and inflexible search of efficient structures typical of NAS approaches. Extensive experiments over two state-of-the-art datasets prove the strength of our framework, showing that it can generate powerful architectures with high probability. Moreover, GNN2GNN outperforms possible counterparts for generating NN architectures, and shows flexibility against dataset quality degradation. Finally, GNN2GNN paves the way towards generalisation between datasets.

Introduction

Designing high-performing neural network architectures has traditionally relied on human intuition or computationally intensive Neural Architecture Search (NAS) algorithms. While NAS has automated much of this process, it remains fundamentally a search problem over a massive space of possibilities. This recursive search is often inefficient, inflexible to changing requirements, and fails to capture the underlying "design principles" that make an architecture successful.

GNN2GNN proposes a paradigm shift: moving from "searching to find" to "learning to build." By framing architecture design as a generative task, the framework aims to learn the latent grammar of high-performance networks, allowing for the generation of optimized architectures in a single step.

Comparison between the traditional recursive NAS approach and the proposed generative GNN2GNN approach.

Methodology

The GNN2GNN framework represents neural network architectures as Directed Acyclic Graphs (DAGs), where nodes represent operations (e.g., convolutions, pooling) and edges represent data flow. The core of the system is a three-way adversarial training setup:

  1. The Generator (G): A Graph Neural Network that constructs architectures by iteratively scoring edges and assigning operations to nodes. It utilizes Gumbel-Softmax sampling to maintain differentiability during the edge selection process.
  2. The Discriminator (D): This component is trained to distinguish between "real" architectures from a benchmark dataset and "fake" ones produced by the Generator, ensuring structural plausibility.
  3. The Valuer (V): Acting as the "reward" signal, the Valuer predicts the performance (accuracy) of a given architecture. Research found that the Valuer performs more stably when treating this as a classification task (Strong vs. Weak architectures) rather than a direct regression of accuracy.
The three-way adversarial framework consisting of a Generator, Discriminator, and Valuer.

By optimizing these three components together, the Generator learns to map random noise to the space of high-performance neural architectures.

Experimental Results

The framework was evaluated on two major NAS benchmarks: NAS101 (423k architectures) and NATS (15k topologies).

Performance and Ranking

On the NAS101 benchmark, GNN2GNN achieved a peak accuracy of 94.32%, outperforming established NAS methods such as DARTS (93.02%), NAO (93.33%), and ENAS (92.54%). Furthermore, the "Best Rank" achieved by GNN2GNN (5372) was significantly closer to the global optimum compared to DARTS (57079).

Structural Robustness

A key finding was the model's resilience to dataset quality degradation. Even when the top 90% of architectures were removed from the training set, GNN2GNN continued to generate models that outperformed the remaining training data. This suggests that the framework successfully captures fundamental structural principles rather than merely memorizing the best-performing examples in the dataset.

Consistency

GNN2GNN demonstrated high generation consistency. Over 80% of its generated models achieved an accuracy greater than 90%, a significant improvement over random search or simpler generative adversarial approaches like MOLGAN.

Conclusion

GNN2GNN demonstrates that neural architecture design can be effectively treated as a generative task rather than a search problem. By learning the underlying design criteria of successful networks, the framework offers a more efficient and flexible alternative to traditional NAS. Future work will explore improving generalization across different domains and integrating GNN2GNN as a proposal mechanism within hybrid search-generation pipelines.

Summary updated

Abstract

The success of Neural Networks (NNs) is tightly linked with their architectural design---a complex problem by itself. We here introduce a novel framework leveraging Graph Neural Networks to Generate Neural Networks (GNN2GNN) where powerful NN architectures can be learned out of a set of available architecture-performance pairs. GNN2GNN relies on a three-way adversarial training of GNN, to optimise a generator model capable of producing predictions about powerful NN architectures. Unlike Neural Architecture Search (NAS) techniques proposing efficient searching algorithms over a set of NN architectures, GNN2GNN relies on learning NN architectural design criteria. GNN2GNN learns to propose NN architectures in a single step -- i.e., training of the generator --, overcoming the recursive approach characterising NAS. Therefore, GNN2GNN avoids the expensive and inflexible search of efficient structures typical of NAS approaches. Extensive experiments over two state-of-the-art datasets prove the strength of our framework, showing that it can generate powerful architectures with high probability. Moreover, GNN2GNN outperforms possible counterparts for generating NN architectures, and shows flexibility against dataset quality degradation. Finally, GNN2GNN paves the way towards generalisation between datasets.
BibTeX
@inproceedings{agiollo2022gnn,
  author = {Andrea Agiollo and Andrea Omicini},
  title = {{GNN2GNN: Graph Neural Networks to Generate Neural Networks}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=BrgGA88scgc},
}
Figure