Autonomously and Simultaneously Refining Deep Neural Network Parameters by a Bi-Generative Adversarial Network Aided Genetic Algorithm

Yantao Lu, Burak Kakillioglu, Senem VelipasalararXiv 2018 · arXiv

Introduction

The design of Deep Neural Networks (DNNs) has traditionally relied on expert intuition and extensive trial-and-error. Determining the optimal number of layers, filter sizes, and architectural components like Batch Normalization or Dropout is a complex task. While Genetic Algorithms (GAs) have been employed to automate this "Neural Architecture Search" (NAS), they often struggle with the "exploitation" of continuous parameter spaces. Standard GAs typically select from a predefined discrete set of values, failing to converge effectively as the search space expands.

To address these limitations, this paper introduces a hybrid optimization framework. It integrates a Genetic Algorithm for discrete architectural decisions with a novel "Bi-Generative Adversarial Network" (Bi-GAN) to refine continuous parameters, enabling the discovery of high-performing, efficient network architectures.

Methodology

The proposed framework operates through a two-tiered optimization process that balances global exploration with local exploitation.

Figure 1: The hybrid optimization framework combining Genetic Algorithms with a Bi-GAN optimizer.

The Genetic Algorithm Tier

The GA tier manages the high-level "blueprint" of the network. It handles discrete architectural choices, such as selecting activation functions (e.g., ReLU vs. Tanh), kernel sizes, and toggling features like Batch Normalization, Dropout, or Max Pooling. This tier ensures broad exploration of the architectural search space.

The Bi-GAN Tier

The core innovation is the Bi-GAN tier, which refines continuous parameters—such as the specific number of neurons in a layer—within a nearly continuous range. Unlike standard GAs that pick from a list, the Bi-GAN "dreams up" precise values.

Figure 2: Detailed architecture of the Bi-GAN component, featuring dual generators and evaluators.

As shown in Figure 2, the Bi-GAN architecture consists of two generators ($G_1, G_2$) and two evaluators ($E_1, E_2$). The generators propose parameter sets, which are then used to build and train networks. The evaluators measure the resulting performance (accuracy). A discriminator then analyzes these results to distinguish between "good" and "bad" parameter sets. This competitive setup provides a stable training signal; the discriminator identifies which generator is performing better, forcing the other to improve.

Experimental Setup

The framework was evaluated using the ModelNet40 dataset, which contains 3D CAD models across 40 categories. These models were converted into $30 \times 30 \times 30$ voxel grids. The system's performance was benchmarked against the hand-crafted 3D ShapeNets architecture and traditional GA-only approaches.

Results and Discussion

The Bi-GAN + GA method demonstrated significant improvements over both manual and standard automated designs.

Method Accuracy (%)
3D ShapeNets (Hand-crafted) 84.17
Small-set GA 82.94
Large-set GA 36.41
Proposed Bi-GAN + GA 85.20

A key finding was the failure of standard GAs in large search spaces (Large-set GA), where accuracy plummeted to 36.41%. In contrast, the Bi-GAN successfully navigated this complexity. Furthermore, the system discovered that precise parameter tuning allowed a shallower 3-layer convolutional network to outperform the 4-layer 3D ShapeNets model. When allowed to search up to 5 layers, the proposed method achieved an accuracy of 86.62%.

Conclusion

By combining the exploratory power of Genetic Algorithms with the fine-tuning capabilities of a Bi-GAN, this framework offers a robust solution for autonomous DNN design. While computationally intensive, the approach moves beyond discrete search spaces, allowing for the discovery of optimized architectures that exceed the performance of human-designed models. Future work may involve scaling this method to deeper architectures and optimizing hyperparameters like learning rates alongside the network structure.

Summary updated

Abstract

The choice of parameters, and the design of the network architecture are important factors affecting the performance of deep neural networks. Genetic Algorithms (GA) have been used before to determine parameters of a network. Yet, GAs perform a finite search over a discrete set of pre-defined candidates, and cannot, in general, generate unseen configurations. In this paper, to move from exploration to exploitation, we propose a novel and systematic method that autonomously and simultaneously optimizes multiple parameters of any deep neural network by using a GA aided by a bi-generative adversarial network (Bi-GAN). The proposed Bi-GAN allows the autonomous exploitation and choice of the number of neurons, for fully-connected layers, and number of filters, for convolutional layers, from a large range of values. Our proposed Bi-GAN involves two generators, and two different models compete and improve each other progressively with a GAN-based strategy to optimize the networks during GA evolution. Our proposed approach can be used to autonomously refine the number of convolutional layers and dense layers, number and size of kernels, and the number of neurons for the dense layers; choose the type of the activation function; and decide whether to use dropout and batch normalization or not, to improve the accuracy of different deep neural network architectures. Without loss of generality, the proposed method has been tested with the ModelNet database, and compared with the 3D Shapenets and two GA-only methods. The results show that the presented approach can simultaneously and successfully optimize multiple neural network parameters, and achieve higher accuracy even with shallower networks.
BibTeX
@misc{lu2018autonomously,
  author = {Yantao Lu and Burak Kakillioglu and Senem Velipasalar},
  title = {{Autonomously and Simultaneously Refining Deep Neural Network Parameters by a Bi-Generative Adversarial Network Aided Genetic Algorithm}},
  year = {2018},
  eprint = {1809.10244},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1809.10244},
}
Figure