An Approach for Efficient Neural Architecture Search Space Definition

Léo Pouy, Fouad Khenfri, Patrick Leserf, Chokri Mraidha, Chérif LarouciarXiv 2023 · DOI · arXiv

TL;DR

This paper introduces a novel, hierarchical, cell-based search space for Neural Architecture Search (NAS) that aims to balance generality and efficiency. By defining a flexible structure for convolutional neural networks, the authors propose a method to discover state-of-the-art architectures more effectively, reducing the computational burden of NAS.

What Problem Does This Paper Address?

Developing effective deep learning models, particularly for computer vision, often requires significant expertise to design optimal neural architectures and tune hyperparameters. This process is empirical and time-consuming. Neural Architecture Search (NAS) aims to automate this, but existing methods often result in astronomically large search spaces or require immense computational resources (e.g., thousands of GPU days). This paper tackles the challenge of defining a NAS search space that is both general enough to discover high-performing, state-of-the-art architectures and efficient enough to be explored within a reasonable timeframe. A key aspect is the need to consider resource constraints, especially for embedded systems, which influences the architecture's execution time and computational demands.

Methods

The core contribution is the definition of a hierarchical, cell-based search space. This approach breaks down the architecture into repeating "cells," which are then stacked in layers. This significantly reduces complexity compared to searching for entire architectures from scratch.

Key methodological aspects include:

  • Cell-based Structure: The search space is organized around repeating cells, inspired by prior work that showed the effectiveness of module repetition. This allows for the creation of deep models without exponentially increasing the search space complexity.
  • Hierarchical Design: The search space incorporates the notion of spatial resolution control, a feature found in state-of-the-art hierarchical NAS methods. This means that within a cell, operations can either maintain, downsample (halving dimensions, doubling filters), or upsample (doubling dimensions, halving filters) the spatial resolution of the feature maps.
  • Convolutional and Reduction Parts: Each cell is divided into a "convolutional part" and a "reduction part."
    • The convolutional part consists of multiple parallel "pipelines," each containing a series of "blocks." Blocks are the fundamental building units, which can be simple operations (like convolutions or pooling) or combinations of operations with options for batch normalization and activation functions.
  • The reduction part handles the spatial resolution changes using downsampling, upsampling, or samesampling operations, often involving merge operations like addition or concatenation.
  • Adaptive and General: The authors emphasize the flexibility of their search space, allowing users to define custom blocks and options, making it adaptable to various needs and potentially incorporating recent advancements like SENets.
  • Optimization Loop: The paper outlines a double optimization loop (cell population and architecture population) for exploring the search space, though preliminary testing used a single loop optimizing directly on the architecture vector. They employed a multi-objective Evolutionary Algorithm (EA) for optimization.

Datasets

The paper reports results on the MNIST dataset.

Key Insights and Ideas

  • Cellular Repetition is Key: The paper reinforces the idea that repeating well-designed modules (cells) is a powerful strategy for building deep and effective neural networks, simplifying the NAS problem.
  • Hierarchical Resolution Control Matters: Incorporating the ability to control spatial resolution at different stages of the network is crucial for extracting finer features and improving performance, as demonstrated by prior work.
  • Flexibility in Block Definition: Allowing users to define custom blocks with various operations and options (like batch normalization and activation functions) makes the search space more adaptable and capable of discovering novel architectures.
  • Balancing Generality and Efficiency: The proposed hierarchical, cell-based approach strikes a balance between the vastness of possible architectures and the computational feasibility of searching them, a critical challenge in NAS.

Key Results

The preliminary implementation of the NAS framework was tested on MNIST. The authors obtained a Pareto front showing a trade-off between accuracy and the number of parameters. The best model found achieved an accuracy of 99.35% with an error rate of 0.65% after additional training, using 15 million trainable parameters. While this didn't surpass the absolute state-of-the-art human-crafted architectures, the results are considered promising and validate the viability of their search space. The method demonstrated steadiness, finding similar architectures across multiple runs.

Limitations and Future Work

  • Limitations:
    • The method, while promising, did not reach the accuracy of the very best recent state-of-the-art human-crafted architectures on MNIST.
  • The preliminary testing used a single optimization loop, which might be less efficient than the proposed double loop.
  • Future Work:
    • Improve the search space further.
  • Enhance the searching strategy to achieve closer-to-state-of-the-art accuracy.
  • Validate the framework on other datasets like CIFAR10, CIFAR100, ImageNet, and Cityscapes.

Summary updated

BibTeX
@misc{pouy2023approach,
  author = {Léo Pouy and Fouad Khenfri and Patrick Leserf and Chokri Mraidha and Chérif Larouci},
  title = {{An Approach for Efficient Neural Architecture Search Space Definition}},
  year = {2023},
  eprint = {2310.17669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.17669},
}
Figure