Evolutionary Neural Architecture Search and Applications [Guest Editorial]

Yanan Sun, Mengjie Zhang, Gary G. YenIEEE Comput. Intell. Mag 2021 · OpenReview

Abstract

This special issue has brought together researchers to report state-of-the-art contributions on the latest research and development, up-to-date issues, challenges, and applications in the field of ENAS. Following a rigorous peer review process, five papers have been accepted for publication in this special issue. The articles included are briefly summarized.

Introduction

Deep Neural Networks (DNNs) have fundamentally transformed the landscape of artificial intelligence. However, the efficacy of these models is intrinsically linked to their underlying architectures. Historically, the design of high-performing architectures has been a manual, "black art" process requiring significant domain expertise and extensive trial-and-error. Neural Architecture Search (NAS) was introduced to automate this design process, yet it faces significant challenges: high computational costs, non-differentiable search spaces, and the need to balance multiple conflicting objectives like accuracy and hardware efficiency.

Evolutionary Neural Architecture Search (ENAS) leverages evolutionary computation (EC)—including genetic algorithms and particle swarm optimization—to address these hurdles. The guest editors argue that EC is uniquely suited to navigate these complex, multi-objective landscapes without requiring deep prior expertise from the end-user. This guest editorial introduces a special issue dedicated to the latest advancements in ENAS, showcasing how these bio-inspired algorithms can navigate complex search spaces to produce optimized models for diverse applications.

Methodological Advancements in ENAS

The papers featured in this issue present five distinct strategies to improve the efficiency and effectiveness of architecture search:

1. Multi-Objective Model Compression

One contribution addresses the deployment of DNNs on resource-constrained edge devices. By employing a two-stage evolutionary strategy involving pruning and quantization, the researchers successfully optimized both model size and accuracy, providing a Pareto front of compact architectures suitable for various hardware budgets.

2. Unsupervised Search Frameworks

To mitigate the dependency on large labeled datasets, the FaUNAE algorithm introduces an unsupervised NAS approach. Utilizing a student-teacher framework and contrastive-loss functions, this method evaluates architectures without requiring ground-truth labels during the search phase, significantly broadening the applicability of NAS.

3. Efficient Performance Prediction

A major bottleneck in ENAS is the computational cost of training candidate networks. To solve this, a novel regression-based performance predictor was developed. By using a graph edit distance-based encoding strategy, the model can accurately predict a network's performance without full training, drastically reducing the search time.

4. Bio-inspired Regression for Time-Series

Moving beyond classification, the DNR-SMS model applies ENAS to "Dendritic Neural Regression." Using a "States of Matter Search" (SMS) algorithm, this work optimizes structures for time-series forecasting, specifically targeting wind power prediction—a critical task for green energy sectors.

5. Self-Adaptive Mutation Strategies

The SaMuNet approach introduces a self-adaptive mutation component to the evolutionary process. This mechanism prevents population degradation and the "loss of experience" typically seen in early search stages, ensuring a more robust and efficient exploration of the architecture space.

Experimental Validation and Benchmarks

The proposed methods were rigorously tested against standard benchmarks and real-world datasets:

  • Image Classification: On CIFAR-10 and CIFAR-100, SaMuNet outperformed 17 competing methods, achieving superior accuracy with lower resource consumption.
  • Computer Vision Tasks: The FaUNAE algorithm demonstrated that unsupervised NAS could achieve performance levels comparable to supervised methods in object recognition, detection, and instance segmentation across four large-scale benchmarks.
  • Energy Forecasting: The DNR-SMS model significantly improved wind speed prediction accuracy compared to traditional forecasting techniques.

Key Insights

The collective research in this special issue highlights several critical shifts in the field:

  • From Accuracy to Efficiency: There is a growing emphasis on hardware-aware design and multi-objective optimization, ensuring models are practical for real-world deployment.
  • Predictive Modeling: The shift toward performance predictors and self-supervised evaluation is making ENAS more computationally accessible.
  • Robustness of EC: Evolutionary algorithms continue to prove their worth in handling the non-convex and discrete nature of neural search spaces where gradient-based methods often struggle.

Conclusion and Future Outlook

While ENAS has made significant strides, challenges regarding search cost and the transferability of evolved architectures to massive datasets remain. However, the advancements presented in this special issue—ranging from self-adaptive mutations to unsupervised search—demonstrate that evolutionary computation is a powerful tool for democratizing deep learning. As these techniques mature, they will become essential tools for the broader adoption of optimized AI across diverse industries.

Summary updated

Abstract

This special issue has brought together researchers to report state-of-the-art contributions on the latest research and development, up-to-date issues, challenges, and applications in the field of ENAS. Following a rigorous peer review process, five papers have been accepted for publication in this special issue. The articles included are briefly summarized.
BibTeX
@inproceedings{sun2021evolutionary,
  author = {Yanan Sun and Mengjie Zhang and Gary G. Yen},
  title = {{Evolutionary Neural Architecture Search and Applications [Guest Editorial]}},
  booktitle = {IEEE Comput. Intell. Mag},
  year = {2021},
  url = {https://openreview.net/forum?id=pD3wnj25Qx8},
}
Figure